The BLS Strikes Back

"You talkin' to me?" That famous line could describe the response of the Bureau of Labor Statistics to a critical NBER study, which suggested that the Current Population Survey's labor force questions undercount the nation's employed and multiple job holders because they fail to capture much of the informal economy—or gig work.

The BLS has fired back, defending the CPS labor force questions in a recent Monthly Labor Review article. In the article, BLS researchers compare results from the CPS with those from the American Time Use Survey, which asks respondents about "income-generating activities" in addition to whether they have a main or other job. The finding: the nation’s overall employment figures are minimally affected by including respondents who report unemployment on the CPS and income-generating activities on the ATUS. This inclusion would increase overall employment by between 0.4 and 3.0 percent. A bigger effect is seen for multiple job holders. By including in the multiple job holder count those who report having a single job on the CPS and additional income-generating activities on ATUS, the increase in multiple job holders ranges from between 3.0 and a substantial 20.7 percent. The increase is especially large for young adults (as high as 42 percent for people under age 25), women (27 percent), and the less educated (23 percent for those with a high school diploma or less education).

While admitting there could be more multiple job holders than the official CPS numbers indicate, the BLS dismisses the notion that the CPS has missed a surge in gig work. The article concludes: "Despite anecdotal evidence of a large increase in the number of gig workers in recent years, ATUS estimates do not show a marked increase since 2003–07 in either the percentage of people who did income-generating activities or in the amount of time spent by those who did these activities."

Source: Bureau of Labor Statistics, Measuring Labor Market Activity Today: Are the Words Work and Job too Limiting for Surveys?

Retirement Plan Participation Declining? No, says EBRI

The percentage of workers who participate in a retirement plan is declining, according to the Current Population Survey. Don't believe it, says the Employee Benefit Research Institute.

In an ongoing battle with the redesigned Current Population Survey, EBRI's Craig Copeland analyzes the supposed decline in retirement participation recorded by the CPS and argues that something is very wrong with the survey's data. Reporting on this problem has become an annual undertaking by Copeland. This is his third analysis since the CPS was redesigned, and no resolution seems to be in sight. The Demo Memo posts about his earlier reports can be found here and here.

According to the Current Population Survey, the percentage of full-time, full-year workers who participate in a retirement plan fell from 54.5 percent in 2013—before the CPS was redesigned to better capture retirement income—to just 41.0 percent in 2016. Among workers aged 55 to 64, participation fell from 57.1 percent in 2013 to 48.1 percent in 2016.

These figures are at odds with rising participation rates found in other government surveys, says Copeland, such as the Bureau of Labor Statistics' National Compensation Survey. Among private-sector workers at establishments with 500 or more employees, the NCS found a stable 76 percent participating in a retirement plan from 2013 to 2016. The CPS found only 47 percent participating in 2016 (after the redesign), down from 64 percent in 2013 (before the redesign). A study of IRS data confirms stability in retirement plan participation rather than the decline charted by the CPS.

"Rather modest modifications could be made within the CPS questionnaire along the lines of other federal government surveys to improve the retirement plan participation estimates," concludes Copeland. "Until that time, any person or organization using the data or those reading analyses from the CPS data need to be aware of the issues with the data. The estimates from the most recent surveys could easily be misconstrued as erosions in coverage, as opposed to an issue with the design of the survey."

Source: Employee Benefit Research Institute, Current Population Survey: Issues Continue for Retirement Plan Participation and Retiree Income Estimates

Americans May Be More Honest in Online Surveys

Phone surveys may be understating financial stress, according to a Pew Research Center experiment. In an effort to determine "mode effects," Pew randomly assigned survey respondents to telephone or online modes when asking them questions about financial stress. Respondents were more likely to report financial stress when answering online than when talking to an interviewer by phone.

"Survey researchers have long known that Americans may be more likely to give a 'socially desirable' response (and less likely to give a stigmatized or undesirable answer) in an interview-administered survey than in one that is self-administered," notes Pew.

Percent saying they received financial help from a relative in past year
Phone: 15%
Online: 26%

Percent saying their personal finances are in poor shape
Phone: 14%
Online: 20%

Regardless of income, Americans are more likely to report poor finances when answering online than by phone, says Pew. "Researchers studying financial stress should consider that phone surveys have, at least to some degree, been understating the share of Americans experiencing economic hardship."

Source: Pew Research Center, Personal Finance Questions Elicit Slightly Different Answers in Phone Surveys than Online

Can Google Street View Determine Local Demographics?

Can Google Street View combined with deep learning-based computer vision provide accurate and up-to-date demographic profiles of local areas? The answer is yes, according to an astonishing study appearing in arXiv, an online repository of scientific papers.

Using 50 million Google Street View images of cars in 200 American cities, the study's researchers determined, with the help of a "machine vision framework based on deep learning," the make, model, and year of each car (2,657 categories). They then used that information to "accurately estimate income, race, education, and voting patterns, with single-precinct resolution." The average precinct has a population of only about 1,000, say the researchers. Here are some of their findings, in their own words...

• "We successfully detected 22 million distinct vehicles, comprising 32% of all the vehicles in the 200 cities we studied, and 8% of all vehicles in the United States."
• "Our model detects strong associations between vehicle distribution and disparate socioeconomic trends."
• "The vehicular feature that was most strongly associated with Democratic precincts was sedans, whereas Republican precincts were most strongly associated with extended-cab pickup trucks."
• "Our estimates accurately determined that Seattle, Washington is 69% Caucasian."
• "We estimated educational background in Milwaukee, Wisconsin zip codes, accurately determining the fraction of the population with less than a high school degree."

The researchers ask whether this type of analysis eventually could replace costly and time-consuming door-to-door efforts such as the American Community Survey. "As digital imagery becomes ubiquitous and machine vision techniques improve, automated data analysis may provide a cheaper and faster alternative," they suggest.

Source: arXiv, Using Deep Learning and Google Street View to Estimate the Demographic Makeup of the US

CPS Redesign Still A Problem

The Employee Benefit Research Institute is still unhappy with the redesigned Current Population Survey. The changes to the CPS questionnaire, introduced to the full sample in 2014, were designed to better capture pension income. That they did, but the changes also resulted in sharp declines in estimated retirement plan participation among workers. The declines are larger than any in the past, notes EBRI, and they are inconsistent with the steady participation recorded by other government surveys.

Last year EBRI examined the problem with the CPS redesign in an analysis of 2014 data (see the Demo Memo post about it here). Now with 2015 data in hand, EBRI finds the problem not only continuing but worsening. Among full-time wage and salary workers aged 21 to 64, the percentage who work for an employer that sponsors a retirement plan fell by a whopping 12 percentage points between 2013 (traditional questions) and 2015 (redesigned questions). Something is wrong with this picture.

Although EBRI admits that pension income estimates are improved by the redesigned CPS, it believes something must be done to improve retirement plan participation estimates. "Currently the U.S. Census Bureau has no plans to revise the CPS," EBRI states. "Rather modest modifications could be made within the CPS questionnaire along the lines of other federal government surveys to improve the retirement plan participation estimates. Until that time, any person or organization using the data or those reading analyses from the data need to be aware of the issues with the data."

Source: Employee Benefit Research Institute, Another Year After the Current Population Survey Redesign and More Questions about the Survey's Retirement Plan Participation Estimates

Even The Best Surveys Have Problems

Just how accurate are even the best surveys—the federal government's Current Population Survey, for example. The CPS is the source of vital economic indicators such as labor force participation and unemployment. Do the rates it measures reflect reality?

A National Bureau of Economic Research analysis examines this question, analyzing some of the basic indicators measured by the CPS, the Consumer Expenditure Survey, and the Behavioral Risk Factor Surveillance System. The goal of the analysis was to determine how much certain indicators vary depending on the number of times survey interviewers attempt to contact respondents. Disturbingly, the indicators vary quite a bit. The labor force participation rate is lower, for example, among participants who respond on the first attempt and higher for those who require three or four attempts before participating in the survey—even after controlling for demographic characteristics. The labor force participation rate was just 63.0 percent among respondents reached on the first attempt and climbed to 72.3 percent among respondents who could be reached only after three or four attempts. The unemployment rate was higher among those reached on the first attempt (8.1 percent) than among harder-to-reach respondents (6.7 percent). The researchers found similar troubling variations in measures produced by the other two surveys: household spending (Consumer Expenditure Survey), and obesity (Behavioral Risk Factor Surveillance System).

How much do our important socioeconomic indicators reflect reality and how much are they a function of who is easiest to reach? What about those who can't be reached after multiple attempts? The authors of the study are concerned.

Source: National Bureau of Economic Research, Difficulty to Reach Respondents and Nonresponse Bias: Evidence from Large Government Surveys, Working Paper 22333 ($5)