Showing posts with label planning research. Show all posts
Showing posts with label planning research. Show all posts

Thursday, May 1, 2014

More Notes on Planning Research: Always Include Your Sample Size, and Stop It with Pie Charts

The American Planning Association just released new survey data about what people want from their communities. The link is here. This is an update to their 2012 survey, which had some interesting claims. However, the 2012 survey was poorly described and I was skeptical of the generalizability from the data. I wrote a post about it. I know the authors of the 2012 report read my previous post because they sent me emails about it. They have improved their research methods not at all in the interim.

My main complaints are that when you present survey data you must include sample sizes, and stop using pie charts. The APA loves endless sheets of pie charts without any data about sample size within each group. It is not hard to include your n. (Here is a link to the only appropriate use of a pie chart.)

The sample size matters for understanding the accuracy of the data. For the entire sample the margin of error is about 2.7 (assuming 95% CI). The data are broken into subsets, though, so we don't know how many millennials or older people are actually in the sample. Millenials are about 25% of the total population, so if they are proportionately sampled that's about 325 people, and a MoE of 5.4% for their responses. That MoE changes the interpretation of the data quite a bit by introducing much more uncertainty of the claims. Uncertainty doesn't lend itself easily to infographics, though.

This Transportation for America survey from a couple weeks ago suffers from similar sample size issues, but at least presents the data in a way that the reader can assess the veracity, and they don't have noxious pie charts. That doesn't stop reporters from gleaning far too much insight from the data. See here, here, here, or just find one of the many other examples. Again with the MoE, though, is that the data sheet reports the MoE as 3.7, yet that is for the full sample of 703, not the subgroups created. As the survey collected 70 responses from 10 different cities the MoE is actually much larger for the data as presented. For any given city the MoE is 11.7% at 95% CI, so the entirety of the data should be used with caution when analyzing subgroups.

So below is what I wrote two years ago, and it stands for these reports again. Analysis using descriptive data can be very powerful if done well, and the difference between doing it well and not doing it well isn't that big.
Reports like this bother me in part because I teach planning research courses and would be distraught if any of my students turned in a report of this quality (without additional explanation, anyway). But the larger issue is that low quality research--whether it confirms or opposes your personal preferences--reduces the signal to noise ratio. Reports like "Planning in America" are noise that cloud our ability to understand critical issues and policy (the signal in this case). At the very least the full methodology should be explained, pie charts jettisoned and sample sizes included in tables and graphs. As for planning research, reports like this are why I argue planning education should focus primarily on numerical literacy and well-crafted basic research with descriptive statistics rather than advanced regression analysis. We should train planners to communicate with data rather than claim to be psuedo-econometricians. Many of the greatest failures of planning can be directly attributed to planners' inability to understand the fundamentals of quantitative data. (See here for an explanation of the most egregious example.) Reports like "Planning in America" make the situation worse, at least as currently presented. Let's not get excited about the claims made in it.

Thursday, October 17, 2013

Long Beach Has Everything. So What's the Matter?



Last week I spent a few days in Long Beach, California at a conference focused on urban freight. Good stuff. I like Long Beach a lot, and always have. But as I was wandering around downtown I was struck by a couple of things. First, where was everybody? Whenever I travel I have to adjust my expectations for traffic and pedestrian activities far away from even my relatively quiet Manhattan neighborhood. Yet Long Beach was empty. I realize this was mid-week in early October, but it was striking, and leads to my second thought: Long Beach has everything. It's like SNL's Stefon was an urban planner for the city. As near as I can tell, Long Beach has at least one example of every major trend in planning over the past few decades:


Plus other trendy things like loft apartments and brew pubs. Yet while I walked around downtown there was hardly any action. Buses park overnight on the transit street (see GIF at top, which was taken at about 8pm on a Wednesday night at what should be a hot part of town). Street life was non-existent. An unusual amount of the retail space was occupied with fitness trainers, which suggests that there isn't much demand for the space overall. And for all of Long Beach's efforts they didn't even make California's Most Livable Cities list!

Chasing planning trends is no guarantee of success for a city. I'm sure everything Long Beach pursued was justified by someone in good faith. However, Long Beach performs somewhat worse than the region in terms of unemployment and other economic indicators. Whatever they have done hasn't been much of a success. All cities should be wary of trendy planning. Perhaps Long Beach can be a case study (this would be a good thesis if any students are reading). Of course, it is harder to predict what is trendy planning and unlikely to be transformative, though there is also a lot of research about the effectiveness of the types of planning efforts Long Beach pursued that gets ignored.