So, could it be that Alfred and Phebe split up soon after the 1881 census and Alfred went off to Glamorgan and had a son Thomas by another woman, or got together with another woman who already had a son Thomas, and Thomas took his surname?
Except that he doesn't have a family with him in 1891.
I suppose you could get the 1899 death cert to see whether it helps at all.