My Fellow Prisoners

What did he mean to say anyway? I don't know. Are we all his fellow prisoners? This is an odd one. Grandpa's going dingy.