So crisp and clear.

Super explanation!

good video, love it

Great . thank you

What a great channel!

@irtizahafiz

2 ай бұрын

Thanks so much!

Thanks

why do we need scheduled job to update cache for every user?

I have one doubt in designing data model. What would happen if We do create separate table POST_USER and include User_id in Post table.

great design and clearly articulate! thanks a lot! i just wonder, why does stream processor needs to talk to feedservice? i thought feed service now just read results from redis cache. could you help clarify?

@irtizahafiz

8 ай бұрын

I should have been clearer. You are right, feed service directly reads from redis cache.

Hi, great content! Why do we need a Post_User table? We could have a UserId column in the Post table that would record an owner's ID ?

@irtizahafiz

Жыл бұрын

Yeah you could do that too. But having a post_user table will let you store more fields about the relationship if needed.

🔥🔥

Thanks for the amazing content! Rather than using a CDC, can we simply write a "post_created" event directly to Kafka from the post service? So the post service does 2 jobs. One, write to the database and two, write an event to Kafka.

@irtizahafiz

Жыл бұрын

Yup! That works too. Totally depends on what kind of architecture you have.

For the Scheduled Job, you said that you will iterate through all the users in your database and update the Feed Cache. For the Scheduled Job, if it updates the Feed for every single user in our system (let's say 5M), would you be adding 5M rows to the Feed Cache? My thought was that the Feed Cache would only store a percentage (lets say 20%) of daily users.

@irtizahafiz

Жыл бұрын

Hi! You can do it both ways depending on what kind of infra you have for database and cache.

For pagination ! Lets assume you have 100 posts cached for each user. Would you consider another service to add more posts to this user cache on reaching last available posts ?

@irtizahafiz

8 ай бұрын

You can store all the IDs in your cache, and paginate there. Given you are only storing IDs, and not post details, you can add a ton there.

3:05 Post_User does not need an ID, the primary key is as a weak entity

You made me to start thinking on a lot of things in my project. Thank you very much! A question to Irtiza or anyone: Step 1) So I fill the Feed cache with new post ids that belongs to a user, that should be displayed for user. Step 2) Probably I should remove the cached posts at a time... But when? When the user saw the post? Or should there be an expiration on each cached post?

@ujwalkumar2374

Ай бұрын

1. Feels like it 2. Both seem valid but second makes more sense. Lets say there is a list of posts for some users but the users have not used the app for a while. It does make sense to remove the post after sometime

One thing missing in the design is about what happens for influencers and celebrities where the Push mode would not make sense.

@vinaymiriyala4522

Жыл бұрын

Do you mind sharing how to account for this op

@HindiCartoonKahaaniyaTv

Жыл бұрын

@@vinaymiriyala4522 use fanout on read model for celebrities. Just before rendering feed fetch all saved feeds along with celebs post which you follow

Why do you need separate ID column for Friends and PostUser table when you can just use composite key (postID, userID - PostUser) which uniquely determines a row?

@irtizahafiz

2 жыл бұрын

I always prefer having an auto incrementing ID column for all my tables. It helps with JOINs in the future, if you are not considering all your use cases right now. And it's worth the performance tradeoff given the simplicity of that column.

there is potential bottleneck on the post api to user-post table before the cdc Kafka. Maybe can partitioning or sharding this part

@jingjingcoming

Ай бұрын

Or swap this part with nosql server

Awesome videos. What is the name of the tool that you used for the diagrams?

@irtizahafiz

4 ай бұрын

Miro :)

Is Moderation service updating Post User table if any post found out to be malicious

@irtizahafiz

9 ай бұрын

Yes that would be the idea. But this design is at a very high level, so I might have not mentioned that explicitly.

what if the redis cache does not have the user id for whom the feed is getting loaded, then the feed service needs to talk to post service? Or will you return no feed for them, which is a poor experience?

@irtizahafiz

3 ай бұрын

Yes. If you run into cache miss, you should always consult the DB with the "same logic".

1.) Why does feedStreamProcessor need to talk to Post service? 2.) How does Feed Service fetch the information of a user whose entry isn't present in cache at all? It should be talking to Friend service, Ranking service and then fetch the relevant details and then push it to cache and return the response, right?

@irtizahafiz

2 жыл бұрын

1. The stream processor will need to pull details of the post. It usually deals with IDs only. 2. Yes, that's correct.

What happens for the posted data when it fails moderation but is still being implemented processes by other workers / has been written into storage

@irtizahafiz

3 ай бұрын

Depending on your tolerance level, you can start processing after moderation, or go ahead and delete records / evict caches after something is flagged as inappropriate by the moderation system.

I have a question. Let's say if A and B are friends. When A creates a post, it writes to the redis cache on server1 to build the feed for friend B. However, friend B gets routed to server2, which means it won't have access to this cache. In other words, if A has 100 friends, and when A creates a post, how do we update the feed cache for these 100 friends? They are in different servers and their cache will not be in server1.

Missing one context, Why feed stream processor interacts with feed service. You were saying "The feed of users". May I know what it is?

@irtizahafiz

4 ай бұрын

The feed is a precomputed set of posts that the user sees on their home/feed page.

if the posts get stored in the CDC before it hits the Modification Stream Processor, then hits the Feed Stream Processor, how is it going to prevent offending messages from being posted?

@irtizahafiz

4 ай бұрын

That's a great point!

👍🚀

would you upload the lecture slide

@irtizahafiz

2 жыл бұрын

Hi! Unfortunately, I don't have the slides for this one. For most of the other ones I started uploading PDFs or slides. Hope that helps!

I don't think storing age just as interger will make sense rather storing dob and parsing that to obtain age at run time is the approach

@irtizahafiz

2 жыл бұрын

Yup! I agree. The purpose of the video was to design the whole system, not dive deeper into individual data model. So I decided to keep things simple : )

@meditationdanny701

2 жыл бұрын

@@irtizahafiz gotcha

api gateways knows which service to hit not load balancer

This is great but I don't think it is efficient to create feeds for users that you don't know if they will use the service at all. On Twitter or other social networks there must be millions of inactive users, that maybe are following Elon Musk, so everytime Elon twits you are doing a lot of unnecessary work for those millions of inactive users. Besides that, I'd like to have more details about the Raking service. On the first example I don't see efficient to get all the post in order to send them to the Ranking service.

@sarang5906

8 ай бұрын

Agreed. It is a trade-off to be made in terms of the freshness of the feed. So one solution could be to refresh the feed only if the user visits and refreshes their Newsfeed Page.

@davidmataviejo3313

6 ай бұрын

It is the price you pay for having the users feed already computed. Users will not use it if they need to wait 1 minute to be ready. And this approach only works for regular users. For users with huge amount of followers, don't follow the same approach

@dontdoit6986

3 ай бұрын

Twitter actually does create feeds for every user with every new post. It’s counter-intuitive, but they do own their servers for performing the compute.