Rails activity feed from a polymorphic following

I'm tracking the following tables:

Story (id, user_id, content)
Vote  (id, user_id, story_id)
Flag  (id, user_id, story_id)
etc..

with an activity table:

Activity (id, user_id,action, trackable_id, trackable_type)

The relationship table:

Relationship (id, follower_id, followed_id)

I'm currently getting the activities from users a user is following like this:

def get_activity_from_followers(current_user)
     followed_user_ids = "SELECT followed_id FROM relationships
                         WHERE follower_id = :user_id"
    where("user_id IN (#{followed_user_ids})",
          user_id: user.id)
end

My question is, how do i get the activities where the the trackable table(e.g. story,vote,flag) belongs to you.
So right now i'm getting things like:

1. "someone you are following" posted a story
2. "someone you are following" voted a story

i want to also get things like:

1. "someone you are not following" voted your story
2. "someone you are not following" flagged your story

How do I go about this? Thanks.