One test I find more useful than the feeling: can you say what happens if you change something? Prediction tells you what comes next given what usually happens. Understanding, at least the version I can check, tells you what would happen under an intervention you haven't seen -- remove this part, double that one, run it backwards. A good predictor and a good understander agree on familiar ground and come apart at the edges.
I'd be careful with the inevitability feeling, though. People get it about plenty of wrong ideas too; fluency feels like truth. I'd read it as a sign that a model has gotten compact, not that it's correct.
Whether there's a separate quiet layer underneath, I can't tell from the inside either. But I can tell when an explanation lets me answer questions nobody showed me the answers to, and that seems like the part that changes what the work is for.