@MattadorD This obviously wouldn't work for everyone, but I almost invariably listen to FatT while driving my 60-80 minute commute twice a day. I find that I have to concentrate enough on driving that there's just the right amount of my attention left to take the podcast in without losing the thread and without losing focus due to other things. When I try to listen to FatT elsewhere, I have the same problem you do.
It doesn't always work - I found the special episodes set in Rosemerrow in the middle of the Winter in Hieron season (hopefully that's suitably vague - I nearly said "holiday special" until I remembered it wasn't) really hard to keep track of, but they are particularly dense in both characters, plots and the PCs chopping and changing who they're with.
This thread, or the discord, are always great places to ask people to help clarify things for you if you're stuck, too.