Ruchika Talwar: Hi everyone. Welcome back to UroToday’s Health Policy Center of Excellence. Today, I’m joined by Dr. Valeska Halstead, who is a recently graduated pediatric urology fellow now transitioning into a faculty role at Seattle Children’s. She’s joining us today to discuss some of her recent work that examines pediatric urology US News and World Report ratings. Thank you, Dr. Halstead, for joining us.
Valeska Halstead: Thank you for having me, it’s wonderful to be here. So, I just wanted to share a few slides outlining the recent survey study that we performed about the US News and World Report rankings specifically for pediatric urologists. And what I want to start off with is demonstrating why we are a little interested in this. There’s been a lot of conversation within the pediatric urology community, both formal and informal discussions, about the validity of the rankings, how they should be used, and everything like that. But I put together a bar chart race that shows the rankings over time since 2009, when they were started, to 2023. And there are a couple of things I want to highlight here. So one is, you can see both the static and dynamic movements of the rankings over time. So those at the top generally stay at the top, but then you see the ones maybe not in the top three—there’s a lot more movement year over year.
But then the second thing that’s very important to notice is when we started in 2009, the number 15 score was at a 37, and the highest was 100. But as we move on through the years, you can see those numbers are clustering even more towards being close to one another. So the number 15 score here is 83, and the top is 100. So either all programs are getting better and receiving higher scores, or maybe some programs are learning how to answer these questions a little bit better. But it’s an interesting finding, just seeing this visually. So our group performed a survey study, an email web-based survey study that we sent out to the Society for Pediatric Urology. And of the 515 people that were emailed, we received over a 51% response rate, which we were very pleased with.
So we feel like we got pretty good representation from everybody. There was representation from all AUA sections, with a good age and demographics spread there too. Seventy percent identified as male, the vast majority completed fellowship, and then the majority also reported working in an academic practice model, but there was representation of all practice models. In a secondary part of our study, we wanted to compare those ranked in the top 10 and those not ranked in the top 10 and see if their answers or responses differed at all. And 26% of respondents did identify as currently working at a top 10 ranked institution in 2022. So we went on in our survey study to ask people what they thought about the metrics: did they think that this was representative of the care patients were receiving, the quality of care that was being given, if those ranked higher than them were providing better care, if those ranked lower than them were providing worse care?
But there are a few items that I want to specifically highlight from the study. We could go on for hours about the other parts, but we’ll highlight just a few here. So, of all of those that were surveyed, 86% said that they would indeed support organized efforts to withdraw from the US News and World Report ranking system. And it’s interesting because this is coming exactly on the tails of all of the law schools and medical schools announcing themselves withdrawing from the ranking system. This is a very high number. This did statistically significantly differ between those ranked top 10 and those not ranked top 10, but even those ranked in the top 10%, more than three-quarters, said that they would support efforts to withdraw.
The second interesting finding we asked is: do you think it’s even feasible for any study to be developed that could adequately distinguish between programs? Is this even a possibility if we decide to stay? More than half said that they didn’t believe this was even a possibility. So if some people would like to stay, but many believe that this just isn’t a possibility to create a new survey, what are our options moving forward? And I like this as a visual representation of some of the responses. So you can see some of the big questions side by side. And if you look at the left side, the green, the blue—those are positive or affirmative responses to some of the questions. And on the right side, the yellow and the orange—those are negative or dissenting responses. And then the far right is neutral. But just on quick glance, you can see that most of the responses are in the negative or dissenting side of things.
So one item we asked was: do you believe people are honest in reporting their data or honest in sharing the results with the US News and World Report? And the vast majority said that they didn’t believe people were being honest. And this is something we heard echoed again and again in the free text responses of our study as well, which again was very interesting. And then we asked if we thought that the ranking system had led to improvements in care. The majority said no. Most were not agreeing with the relative weights or the individual metric items used in the survey. And then we finally asked: do you believe that these are reflective of quality of care? Do you think it reflects patient outcomes? And again, we see that most people do not believe that this is reflective.
And so, if these are our findings, we’re the ones intimately familiar with the ranking system, with what this means, with the medicine behind it. And our patients are seeking skilled inpatient care, and they want to know that they’re getting the best. And so they look to something, and how can we have our patients and their families relying on this system if even those most familiar with it are questioning the validity of it and the value of it as well? Thank you for letting me share our study.
Ruchika Talwar: Really interesting findings here, and I think this is just part of a broader conversation that’s been happening in healthcare in general. You alluded to the fact that law schools and medical schools have withdrawn from US News and World Report rankings, and there are a variety of rankings out there. There are Doximity rankings for residency programs, there are rankings for physician offices, there are rankings for large hospitals and health systems. And you’re right, a tremendous amount of resources goes into figuring out how to improve your score, which may not necessarily reflect a true difference in quality. So I’m curious, as we look to encourage our patients to make empowered healthcare choices, what is your opinion on seeking out Centers of Excellence or specialty care, particularly for very complex pediatric urology conditions?
Valeska Halstead: Yeah, I think what you said is the key point there for complex conditions. I think so easily our patients look to these scores and think, “Oh, I need to go to this place for everything.” And there are very simple procedures that we do in pediatric urology. Sure, they’re nuanced, but they’re simple. And I don’t think patients need to be seeking out Centers of Excellence for this because generally, they’re done very well everywhere. But for certain conditions, for the very complex things, there should be some sort of designation for Centers of Excellence. But this 1, 2, 3 ranking system, I don’t think it really gets at the true difference between programs because you saw the numbers there: number one’s ranked 100, number two’s ranked 99. Is there truly a difference between the two, or should we just say, “Okay, all this group of hospitals are good at this procedure, are good at taking care of this complex problem”?
So I think it’s having those conversations with our patients, which I think a lot of people are already doing, but we need to find a way to get that messaging across more globally when they’re just sitting at home Googling things.
Ruchika Talwar: Yeah, I think it’s a tricky game because although almost everyone has a negative view of the survey, and over half of the participants are not interested and would actually want to withdraw or develop some kind of new measure, everyone’s sort of stuck in the game. And as you alluded to, for some of the simpler procedures, perhaps care can be done locally. So what is your advice to urologists or people who may be referring for pediatric urology conditions, because they may also look at these rankings to figure out where to send patients to you who have sort of complex issues. Tell us, what should the broader urologic community be aware of in the context of your findings?
Valeska Halstead: Yeah, I think that what we heard in the free text responses and what I’m trying to do… I don’t want to give across just my opinion. I’m trying to say, “Okay, this is what we’ve found.” But in the free text responses, the majority of pediatric urologists feel that people are trained well. We go to the same fellowships, we graduate, we have these metrics we have to meet, and people go out and they’re able to do these procedures. But for the more complex, the rare conditions—people know what these rare conditions are—okay, that’s worth sending to a tertiary referral center or having those conversations. I think that’s the way to move forward; it’s also reassuring patients.
I think it becomes a very sticky situation when we have these centers that are at the top, and patients are wanting to seek those out. But for patients who can’t afford it or don’t have access, that’s creating a lot more problems. And that is an entire secondary thing that we could talk about for hours. But I think reassuring those patients that they are getting great care.
Ruchika Talwar: Yeah, I think you’re right. Alluding to healthcare costs, the burdens associated with travel, needing to take time off of work because obviously pediatric patients require their caregivers to accompany them. So I think you’re hitting a lot of important points here. And I liked your study because I just thought it was one of the first pieces of data that we’re seeing linked to a problem that we talk about a lot. So thanks for taking the time to share your findings with us here at UroToday. And we’re always interested in probing these kinds of thought-provoking questions that go down the quality of care and health policy route. So we appreciate your insights.
Valeska Halstead: Wonderful. Thank you so much for having me. This was so much fun.
Ruchika Talwar: And to our audience, thanks again for joining us. We’ll see you next time.