Smart said one of the things that ultimately pushed him towards leaving was the departure of the athletic director who hired him, Norwood Teague.
He also laughed about rumors he was leaving for jobs elsewhere, UCLA among them, in the aftermath of a Final Four appearance.
"It's funny - I think I might have told you this," Smart tells Gottlieb during the podcast. "There's no way I would have left after that. Like, immediately after. There's no way.
"And I remember I had a couple of assistants that were mad at me about that. And then the years after that, I told you about the two guys that came in, Treveon Graham and Briante Weber, there's no way I was leaving those two guys specifically. Never."
He said the circumstances of Teague's departure stung, though.
"I'll say this: I probably would still be at VCU if our athletic director hadn't left me under the cover of darkness," Smart said.
He added: "So, Norwood Teague, who, you know, I still communicate with and I really appreciate. He obviously went through, and did, some dumb stuff at Minnesota. He hired me at VCU, and I was the third assistant at Florida.
"I mean, you talk about sticking your neck out. I really, really benefited from being part of the same coaching tree that Anthony Grant was part of. I mean, let's be honest, that's a big part of why I got the job.
"And we, as AD and coach, had a really, really, really special relationship, kind of like brothers, where we would go back and forth, and we fight sometimes, but we will always have each other's backs.
"And he just had a unique understanding, I think, because he had been a part of a college basketball coaching staff - what the plight of a coach is - and, you know, we talked all the time about, like, 'I'm not going anywhere if you don't go anywhere.'
"And I'll never forget. I got a call from our president, Dr. (Michael) Rao. I was on the road recruiting. It was like, I don't know, [2012], and he said, 'What's going on with Norwood?' And I said, 'I don't know what you're talking about. I'm in Pittsburgh at an AAU tournament.'
"And he said, 'I just got a call from Minnesota, and he's leaving. And that's how I found out. ... And that was really, really hard for me. And so I probably never would have left VCU if he wouldn't have left."
Smart left three years later, after the 2015 season. True to his word, he didn't leave on Graham and Weber, but he said after their senior seasons ended in injuries and a disappointing tournament loss, he was vulnerable to "dark" thinking.
"I was like, so emotionally bothered that that team wasn't able to go do what I thought we could have been able to go do had those guys been healthy and had everything gone right, and that just really bothered me," Smart said. "It really upset me. But the reality is, you've got to get over that. It's not like it only happened to you. It happens every year, all around the country.
"I was just down, I was upset that we lost. You know, you're mad at yourself a little bit. You're just down. You're losing some guys. And you know, you can see other things going on. And, you know, it's just easy to kind of see the glass half-empty at that point.
"But, you know, again, the benefit of time since then is the glass was full."
Smart is currently the coach at Marquette, which is leading the Big East. Smart has been named as one of the coaches to watch for the national coach of the year award.
PHOTOS: 10 years ago, VCU started their run to the 2011 Final Four