My conversation about phones and mental health with Hunt Allcott

Allcott is strong supporter of Haidt, but thinks "causal evidence is going to be hard"

For the post I did a few weeks back on teen mental health and social media/cell phones, I discussed a study where pouches from a company called Yondr were used to lock cell phones in schools. The study showed effects on the measured parameters of student learning and engagement that were lower than one of the authors, Tom Dee, expected. Dee called the results “sobering” and his full Q&A is here.

I also talked to Stanford researcher Hunt Allcott who was a coauthor on the Yondr study. Allcott has done extensive work on social media and mental well being. He has a more upbeat perspective about the data perhaps than Dee and is refreshingly rigorous in how he discusses results.

Here’s my full Q&A with Hunt.

Holden Thorp:

So I’m interested in this because for one thing it’s an interesting topic. For another, you and I have a connection about this because Science published the papers from the big Facebook project that you were part of. That was complicated, and we spent a lot of time trying to figure out how we were going to disclose the right data to support that study. And I think the other thing is that I’m very interested in what happens when somebody with a huge platform puts a stake in the ground about how things are going to come out and then lots of researchers have to deal with that. And there are many examples in history where that hasn’t worked out so great. So I’ve been following this since Haidt’s book (The Anxious Generation) first came out, which is around the same time we did that Facebook project of yours.

I want to start with this Yondr study and then I’m going to ask you about some of the other social media stuff you’ve done if that’s okay. I guess my first question would be tell me what you think the positive findings are in your latest NBER preprint.

Hunt Allcott:

So as you know, we study the effects of schools adopting Yondr smartphone pouches in middle and high schools in the US. And I think maybe broadly the online positive conclusion is that the effects of smartphone pouches on student outcomes are broadly small.

Holden Thorp:

And were you expecting them to be bigger?

Hunt Allcott:

Was I expecting the effects to be bigger? It’s a good question. I’m not sure that I had a strong prior. I do think my own personal experience as an educator — I teach class like other university professors and I try to keep my students from using their smartphones in class because I think it’s a distraction — so my intuition was and frankly continues to be that getting digital distractions out of students’ hands will improve their academic outcomes at least somewhat. So then you might ask, how does that prior interface with the results of our study? My own personal view is that there are perhaps two ways to adjust my prior. One is just that maybe there are not very large impacts on student achievement. Let me say it this way. Perhaps the effects on student achievement are not as large as many educators would predict. The second potential conclusion is that there are other dimensions of student academic achievement other than standardized test scores, which we do not measure in our paper.

Holden Thorp:

Right. Tom Dee told me he thought the results were sobering. Would you agree with that?

Hunt Allcott:

I don’t agree or disagree. I think it just is a matter of how you define the word sobering. I think my previous statement about my priors and the interpretation probably covers my reaction and probably would interface with Tom’s comment.

Holden Thorp:

Okay. And so there have been a lot of studies on time spent on social media that you and others have done that also produced very small effects, sometimes not a lot at all and sometimes something. Do you think this finding is consistent with all of those?

Hunt Allcott:

I think there is a collage of different types of evidence that we need to put together on effects and effects sizes on the different possible outcomes that social media and smartphones might affect. So I’m not sure that I would say these results are inconsistent or consistent with our previous results writ large, but I would say they’re all part of a collage of evidence. The reason I’m sort of saying it this way is that we’ve done randomized experiments, which have a four to six week time horizon, four to six week treatment period, and that has the clearest causal interpretation because it’s a randomized experiment, but the duration of the treatment is very small. And so what I’d really like to know is if you get kids to stop using their smartphones or social media for six years as individuals or maybe as entire social groups of students, how does that impact outcomes? And the school level treatment is longer term. It’s probably not as 100% effective as our deactivating Facebook experimental treatment and its group level. And so because it differs in these different ways, I’m just not sure that I could line up all the effect sizes and say that something is consistent or inconsistent.

Holden Thorp:

Well, let me ask it more directly. Haidt has said that the phone-based childhood is the major cause of the adolescent mental health crisis. Do you agree with that?

Hunt Allcott:

I don’t know.

Broadly, I found his book to be a valuable addition to the discussion and broadly a thoughtful book. And in my interactions with Jon and his team, outside of my reading his book, I have found them to be thoughtful and careful and I’ve learned a lot from the work that they’re doing. Would I myself be positioned to have written that book? I don’t think so, at least not based on my own personal work. I think our work, some of the stuff that you’ve edited and then some of the other work we’ve done, including the paper that came out last week, I think is particularly valuable at providing causal evidence on the effects of smartphones and social media. I don’t think you could go from that alone to Jon’s book, but that’s the value of what Jon is trying to do. He’s trying to combine the causal evidence that our group has provided with many other types of evidence to try to paint a picture of what’s happening and that’s what I think is valuable.

Holden Thorp:

But do you think that the success of it has made it harder for the public to see all these nuances, including the ones that you’re working on that underlie this whole thing?

Hunt Allcott:

I don’t know if there has been a causal effect of the book on the reception of our work other than to say I do think that Jon has drawn a lot of attention to the issue of smartphones and social media. And so actually I think it’s possible that the paper that we released last week got more attention because Jon has drawn attention to this issue broadly than it would have without Jon’s efforts.

Holden Thorp:

I’m confident that’s true.

Hunt Allot:

I didn’t follow all the details of how every individual is responding to the paper that we released, but I found the media articles that I have read in The New York Times and others to be broadly thoughtful and reasonable in presenting the work that we did. So I have no evidence that Jon’s book has made it harder for society to see opposing views or to see evidence that smartphone bans in schools, for example, don’t have the large effects that some might hope.

Holden Thorp:

But I guess my question is if social media and phones are as powerful as, and it’s not just him but lots of people who agree with him, are saying in terms of the mental health consequences that they might have, I mean, shouldn’t we be as scrupulously careful with our messaging as we can possibly be?

Hunt Allcott:

Your question is, if smartphones are as bad as Jonathan thinks, shouldn’t we be scrupulous in the messaging?

Holden Thorp:

Yeah. I mean shouldn’t we go out of our way not to overstate what the effect is until we have a lot of science? I mean, his answer would be that the problem is so bad that — and he told me this on the record when I first started writing about this — that the problem is so bad that it’s a good idea to go ahead and launch this movement before all these academics have done all their causal analysis and everything because it’s hurting people and [he’s] stopping that from happening. And the harm of doing that is not significant. Now, there are plenty of people who say that there are harms from creating this massive movement. And I guess my interest in the harm of that is that if a lot of people are convinced that this is the main reason why we’re having an adolescent mental health crisis where test scores have been going down, and then they see your study and there’s no effect — and I get there’s tons of caveats there and that you would prefer to do your study many, many different times and have all kinds of different permutations to it — but if the public starts seeing, “Oh, we did this and there wasn’t any effect,” I mean, isn’t that going to make the public less trustful of science in this area to begin with?

Hunt Allcott:

I see. Your worry is that a scientist like Jon getting out in front of scientists without the most ironclad evidence will discredit science if it turns out that he’s wrong.

Holden Thorp:

Yeah. Or even if it turns out that he’s 80% right, but the other 20% is really important.

Hunt Allcott:

I guess here would be my view. I think that something is going wrong in Western societies as measured by surveys of subjective wellbeing and some other outcomes. It’s important that we figure out what’s causing that decline and generally that we do everything we can to help our societies do better. And I think smartphones and social media have been one of the leading hypotheses. I don’t think that we will soon reach a point where we have ironclad causal evidence on what’s causing the degradation in mental health. And so in the absence of ironclad causal evidence, it may be beneficial for us as a society to act on our best guess as to what should be done, maximizing expected net benefits. And I think Jon would say, if I’m right that smartphones and social media are quite harmful, then banning them, effectively reducing their use and being smarter about their use, will have potentially large benefits.

If he’s wrong, then we kind of go back to where we were 20 years ago before this stuff when I grew up, and it’s true that we would miss entertainment value. If you restrict the use of something that is useful or entertaining, that’s bad for its users, its consumers.

So being wrong entails a loss, making policies to restrict smartphones and social media if Jon is wrong will entail a loss, but I think a policymaker should act to maximize expected net benefits. And I do think that Jon’s book has helped to clarify many people’s thinking and make a strong argument. And so I think that’s been useful in helping us all evaluate the issues.

Holden Thorp:

Yeah. Okay, good.

Hunt Allcott:

Maybe it’d be useful if I explain why I think the causal evidence is going to be hard.

So I think our randomized experiments (such as those in Science) were awesome, but as I said, they’re individual level and they’re only a few weeks long and so far they’ve been only in the context of elections and so you have these questions as to like what are the effects on wellbeing generalized to longer periods to group level deactivation to 2026 outside of an election. Some of those questions I think I’m more concerned about than others, but there are questions. Then you have our school smartphone study that we released last week, but that’s like a ban on smartphones during school and it’s not all the outcomes, as I said earlier, there may be other student outcomes we’re not measuring. And the teachers love it. Honestly, if I were the principal at a school, I would buy Yonder pouches.

Holden Thorp:

Yeah, I probably would too. I mean, I don’t like it when devices are in my classroom.

Hunt Allcott:

So that’s a case where I wrote a paper saying the effects of Yondr pouches seem to be small, but yet I would still, as the decision maker, implement some phone ban policy such as Yondr pouches.

Holden Thorp:

Yeah. I mean, you’re probably telling people not to pull their devices out in your classroom, right? So you’re effectively doing the same thing.

Hunt Allcott:

Exactly. And especially at the K-12 level, the teachers would probably rather not have to enforce that class by class as opposed to like a school-wide programmatic ban or a Yondr pouch type enforcement mechanism.

The next round of causal evidence that will come through is on the youth social media bans, but I don’t think that’s going to answer all the questions. I think in Australia there’s a huge amount of non-compliance with the ban, people evading the ban. I’m excited to do work in other countries to see how that plays out.

So I think we’ll learn some things there, but then we’ll only learn what are the impacts on 15 year olds on certain outcomes that are measured. And in any non-randomized study, there’s always a concern about confounding. And so I think we’re going to continue to get a collage of evidence and it’s going to get a little bit stronger over the next couple of years, but we’re also going to have to rely on other forms of evidence other than just large scale causal studies done by mostly economists trained in causal inference techniques.

We’re going to have to rely on what teachers say in surveys about whether they like smartphone bans in schools. We’re going to have to rely on what parents see in their families with their own eyes or what they tell us on surveys when they get smartphones out of the hands of their kids and make their kids wait until eighth grade or later to get smartphones. We’re going to have to rely on all sorts of non-causal evidence. I think that’s part, again, of the value I see in Jon’s book is providing this collage of evidence, and I think we’re going to need all of it because the causal evidence is never going to be perfect.

Holden Thorp:

So let me ask a very specific question about social media. So you’ve studied Facebook and Instagram, but what about YouTube? I mean, isn’t YouTube the most popular social media now and how do you ban that without stopping people from watching? And I’m disclosing that I’m on the board, but what about PBS Kids or other TV kind of material that would be good for them to watch?

Hunt Allcott:

I think this is such a hard question. I think that we probably need a partnership among governments and firms and civil society to have effective change here. Because what a government could do is a government could say, “Well, we require age verification subject to some safe harbor provisions for certain set of apps.” And then while you get other apps and you get kids circumventing whatever age verification is safe harbored. And so at this point, I kind of wonder if you really need the platforms to want to do as much as they can and you see things like child accounts and I think if those were designed well, you could imagine YouTube having a child account that allows PBS and doesn’t allow less good content and has a screen time limit on it.

You probably need the smartphone makers themselves to make truly good parental controls and screen time controls so that I can’t just spend an hour on YouTube and then spend my hour on youth Instagram and then spend my hour on youth TikTok and still add up to a lot of hours staring at my screen. I think it’s going to take a collage of work and I think some of it’s going to be very hard to regulate. It’s going to have to be that the platforms are going to want to help probably even more than they already do.

Holden Thorp:

But how does YouTube separate YouTube shorts from, I realize this is a very specific question, from long form video that for kids who don’t have resources and are at home might be good things for them to be watching.

Hunt Allcott:

I think it can’t be done perfectly, but I’m pretty confident that a combination of the bright folks at technology companies could figure out the right way to make sure that kids don’t spend too much time on any individual device and that the content on that device is broadly appropriate. So I think YouTube specifically could create a child account that the smart folks at YouTube have decided is appropriate and I don’t know, they may have already, you probably know better than I do and if you don’t like exactly how that’s been done, I’m sure it could be tweaked. So that’s for the content. And then the overall time I think requires a combination of parents monitoring their kids and the device makers, Apple and Android, making even more useful screen time controls so that parents can say like, “Oh, you spent your hour on YouTube and that’s it, “ as opposed to you got your hour on YouTube and an hour on all of the other apps that are trying to limit themselves individually.

Holden Thorp:

This is great stuff and I appreciate it. The point of my article is just to explain that there’s all these subtleties here. I believe most people in the zeitgeist think the phones and social media are evil and we already know that and it’s scientifically proven, but the science here is much muddier and more complicated than that.

Hunt Allcott:

I agree the science is uncertain. Can I share two other things before we have to go?

Holden Thorp:

Sure,

Hunt Allcott:

One is I have been impressed with Jon Haidt. I mentioned this earlier. I’ve been impressed with Haidt’s interest in the details and the science. So I don’t know if Tom told you this, but after we released our paper, Jon and his collaborator, Zach Rausch sent us a detailed email with a bunch of questions.

Holden Thorp:

Yeah, I bet they did.

Hunt Allcott:

And they were good questions and we’ve answered some of them. We’re working on some others and he’s read our paper as carefully as anybody I’m aware of outside of our collaborator team. I really appreciated that and I appreciated the good questions they’re asking and his famous wiki of evidence about smartphones and social media, I guess there’s several wikis now. I think that’s another example. So I’ve been impressed by the thoughtfulness and then as I’ve read his blog, I’ve been impressed by the thoughtfulness and like what if I’m wrong and this sort of stuff.

So that’s one thing. The other is that I have read many papers and meta analyses of papers that confuse correlation with causation, specifically with regard to social media use and mental health. So obviously as you know well, if you run a survey and correlate people’s social media use with mental health, it could be that more social media use causes worse mental health. It could be that worse mental health causes people to retreat into social media. So there could be reverse causality. It could be there’s some third factor that causes both. It could be that the causation is positive and the other correlations are negative. And so you get a negative correlation, even though the causal effect is positive, you could get any configuration of results. And basically the correlation studies about social media and mental health and in many other areas really tell us very little, perhaps nothing.

And so I have read some number of meta analyses that say, “Oh, well, we’re not sure because there are 73 correlation studies that show positive correlation and 54 studies that show negative correlation. And so who knows?” And my answer would be, who knows, but not because the studies go in different direction, but because they’re correlation studies to start. And so that’s why we did this work on the causal stuff. So I would encourage people to discount statements that are made on the basis of correlation studies in this space. And many of the statements that I’ve made based on correlation studies are statements such as we just don’t know because the correlations go in all directions.

In social science, all relationships are noisy because there’s like a ton of things that cause people’s mental health, their background, how they’re doing from day to day, like whether they got hurt playing basketball and possibly how much social media they use. And what we really care about here is acknowledging that there are many things that cause mental health to get better or worse. What is the slope of any relationship? And so that’s what our causal studies, our randomized experiments identify. So I think you might just, as you read some of the stuff out there outside the economics space, you might just be on the lookout for that confusion.