Note: This is from an earlier 2018 blog post at BrianWansink.com
Two years ago, I had no real idea what a "retracted" journal article was. I had never thought of it, or known anyone who had had one. If someone would have asked me, I would have probably said, "There must be something about it that was wrong."
Two years later, I've had a some journals ask me to rerun analyses and to show original paper-and-pencil surveys from as far back as 2000. When we haven't been able to do so (didn't save the hard-copy surveys or can't track down the coauthor who did the analysis), some of the papers have been retracted because they can't be fully validated.
Since then a few people have asked me if they can believe Article X or Y because they were retracted. I absolutely believe in these articles. The general conclusions are the same, and even if the original hard-copy surveys and original analyses done a number of years ago can't be found, new reanalyses (often done with different statisticians) generate similar conclusions.
On Monday, one of my coauthors pointed out an insightful article that Dr. Stephen Holden had written for The Conversation website titled, "Retraction of a Journal Article Doesn't Make It's Findings False."
One of his many points is that “Retractions may be important signals of reduced confidence in a finding, but they do not prove a finding false. This requires replication.” Social science moves forward because of replication and not rereanalyses.
The Gold Standard of science is when independent researchers can run about the same study and come up with similar conclusions. This is why I wish I had published more detail in the method sections of some of my papers. Instead, when the editor said “Cut out 1000 words,” we cut it out of the methodology section or the references and stayed focused on the idea. Today, you can include all of the methodological detail and post on an online appendix if there isn’t room in the paper.
Looking back, I’d also archive all of the analyses of my coauthors instead of assuming they’ll always have them on their computer or assuming they’ll never be needed. Coauthors can lose laptops, change computers, or leave academia.
As for saving the original pencil-and-paper copy of surveys and coding sheets . . . do so if you have your own warehouse (the photo above was taken in 2005 in Illinois in what was supposed to be my garage). Otherwise you can collect it in a retrievable electronic form (if you think electronic collection won’t affect the results).
Academics is an incredibly enriching calling. As you move forward in a happy career, I hope this is the only time you have to read about a retraction.
There once was a legendary, larger-than-life professor. If Marvel’s 137th movie of this year is named UltraProf, it would be based on John Shank. He taught a dry subject (think Accounting), but his charisma and his orchestration of his class made each class session seem like 60-person David Mamet plays. Every class had passion, drama, and some surprising reveal at the end that people still talked about ten years later. Here’s a quote:
As a teacher John was at home at any level, and always brilliant. I could offer the testimonials of others, however, what brought his classroom performance home to me— and it was a performance in the truest sense of the word—was watching him at an Accounting Round Table at Pitt’s business school. He held 50 top financial officers in the palm of his hand while he presented his material. They were busy individuals with many things on their minds and schedules, but not a one left until John had answered their last question. I can assure you that their staying until the end was not out of courtesy (Bernberg 2008).
Although the opposite is true, some believed he was only about style. This is because he wore Brooks Brothers braces, walked with a MVP swagger, drove a Dartmouth green sports car, and he had a runway model wife who was like some VP of Finance somewhere. His office was professionally-decorated with French draperies, super-thick Dartmouth green carpeting, and a massive 18th century French desk which sat in the middle of the room so his desk chair could face the doorway. Even his two huge perfectly groomed dogs were effortlessly well-mannered. On Saturdays he’d come to work, and they’d sit on either side of his desk and face the door. They were like lions on either side of a throne, and he was like Odin . . . or John Wick. His dogs keep eternal vigilance. My dog wets on me and then licks my face.
This was 1992. Because he had about the highest MBA teacher ratings at Dartmouth's Tuck School and I had about the lowest ratings, he let me sit in on his classes so I could suck less . . . so I could learn better teaching strategies and classroom management skills. One Saturday during a Tuck alumni reunion, I stopped by his office and told him I had overheard some alumni who were still talking about what they had learned in a class they had taken with him 10 years earlier.
He looked up over the top of his half-glasses, and said, “That’s what they’re supposed to do. It means I’ve done my job.”
He said his goal isn’t to teach students to get a great first job (or to, analogously, get a high score on the GRE or MCAT), his goal is to teach them to succeed for wherever they will be in 10 or 20 years. Although he got outstanding teaching ratings, he brushed them off by saying that teacher ratings mainly measured the moment – they mainly measured the warm feelings a student had at the time. Ratings might capture style (which he was very good at), but they may not always measure long-term substance.
Last week, the school year ended. A lot of amazing teachers will take their course evaluation ratings and use them to improve their classes for next year. John’s view was that we need to also focus on the long-term impact of our courses.
I regret that I never had the presence of mind to ask him how he did it -- how he knew what long-term impact to aim at. Since he was on boards and did a lot of consulting with upper management, I suspect he taught his courses like he was teaching board members and upper management. That is, when he was teaching, he treated them like they were high level managers. That’s one way to do it.
A second way to try and teach for long-term impact might be to ask. After they graduate, it will be more apparent to them whether your course helped them live a better life (more useful, meaningful, successful, or whatever), and how your course might be improved. It’s easier to get this feedback than you might think. You’ve probably saved your class lists (somewhere) from 5 years ago. You might have their emails, or the alumni office will have both their emails and their snail mail addresses. You can simply ask them.
[As an example of what your survey could look like, here’s a version of one I sometimes send out to former students a couple years after they graduate. It asks them what they remember, what they found most useful, what they wish they’d learned, and what work-related anecdotes they might have. It also lists the class sessions, and it asks them to circle their 3 most useful class sessions and to cross out their 3 least useful. Sending the right type of email will help you get a great return rate. The first page of the download has a bunch of wording you can use in your email or cover letter, and the second and third pages have example questions you could adopt for your courses.]
Ten or fifteen years after I left Dartmouth I was in Boston, and I rented a car to drive up to visit John. I wanted to thank him for being so generous, and I wanted to prove to myself that his office, desk, and dogs were as amazing as I remembered them. There was a different name on his door. I was too late.
I love the idea of trying to teach for a long-term impact. It’s like trying to create long-term memories. I sometimes think I can remember everything John said to me because he was always so intentional with every conversation. Just like he was with his classes.
At the next reunion, if his former student’s aren’t talking about what they learned 30 years ago, they’ll be talking about how hard he tried. That itself was a great lesson.
Long-term Impact Survey for Class
Academics Only is a how-to community that helps us share our best practices as PhD students, new professors, and independent scholars.
Helpful tools and tips on how to graduate, get tenure, teach better, publish more, and have a super rewarding career.
Some Older Posts