I did a bit more poking around with reading lists and viewable interactions. Once someone is on your recent readers list they are stuck there till they get pushed off, even if they remove your story from their list or move it to a hidden list, anything they do to it in that hidden list wont show up though.