Forget searching mutual connections, when will there be a database for me to find my eskimo brothers? • Taco Corp's EBDBBnB - ...
Жүктеу.....
Пікірлер: 24
@ashishgaonker16 күн бұрын
isn't count 500 X 500 = 250K ? which is 10 time more. Or are we assuming only 10% of friends will be mutual or something like that?
@jordanhasnolife5163
16 күн бұрын
Oof yeah good catch. Point is, fan-out probably won't work here.
@PoRBvG4 сағат бұрын
Thanks for the great content again! Question: In your final diagram, the middle flow (new connection service) shows two layers of Kafka (a stateless consumer in between). Why do we need both layers? Can't the "new connection service(s)" directly push to the corresponding Kafka shard and avoid having the Kafka layer and the stateless consumer?
@ichthyz12 күн бұрын
When adding mutual connections from the Flink nodes. How is it known that the new mutual connections are not already direct connections? e.g. For 10: 3, 4, 15 you are creating 3,15 and 4,15. What if 3,15 and/or 4,15 are direct connections? These connections could also be on a different Flink node/partition.
@jordanhasnolife5163
12 күн бұрын
Fair point - you can always just hit the database first here. We will have a connections table sharded by user Id so we know where to look.
@ishallwin24
12 күн бұрын
Same doubt
@marcgentner132216 күн бұрын
I have a question on brokers and message queue. Do i setup the broker on a server and then set the consumers on other servers? Lets say i have a mail server and i need to classify the emails and send them after classification to there right system. Where do i host the broker and the Ai classification model?
@jordanhasnolife5163
16 күн бұрын
I mean you can technically set them up wherever, but ideally different containers yeah
@SWEcodes17 күн бұрын
Awesome , great video🎉
@truptijoshi25357 күн бұрын
Does profile update mean updating the latest job or education? If yes, why do we need to update the mutual connection DB for that?
@jordanhasnolife5163
6 күн бұрын
Yes - because the data is denormalized in our mutual connections database
@cattnation625711 күн бұрын
Keep doing its help us out so much
@cattnation625711 күн бұрын
You are great bro
@lalasmith213714 күн бұрын
hey, i have some questions if anyone can please help me :) 1) when jordan says shard the database by userID, it means shard it by the hash of the userID (for consistent hashing)? 2) sometimes i see the term partitioned by instead of sharded by, are those the same?
@jordanhasnolife5163
14 күн бұрын
1) yes 2) I think so, others seem to disagree
@lalasmith2137
14 күн бұрын
@@jordanhasnolife5163 thank you so much for taking the time to answer :) also, can't thank you enough for all the knowledge i gained since finding your channel
@alphabeta64416 күн бұрын
Thanks for making this video Jordan. I have two questions: a) You mention "Mutual Cache table", but it appears you are using SQL db for that. Does not cache mean keeping in memory? b) It is mentioned that we need very fast reads ("fast as humanly possible"), should it not engender use of mongodb or something liek that instead of SQL db?
@jordanhasnolife5163
16 күн бұрын
Cache doesn't inherently mean memory, it just means having the result of a computation easily accessible. Why are mongoreads faster thansql?
@fanzhang590313 күн бұрын
Hi Jordan, loving this video. A couple of quick questions: 1. For the adding a connection workflow, is it supposed to be real-time processing or batch? 2. Let's say B accepted A's invite to connect and A wants to view the change right after it, how can we ensure that? 3. Does it make sense if we put the mutual connection data in memory cache servers and have a graph db to store the raw connections so that we can rebuild the cache if any node fails? Any idea or discussion is appreciated. Thanks!
@jordanhasnolife5163
13 күн бұрын
1) Realtime 2) You could first write to a table before using CDC to sink to Kafka and then see the first degree connection there
Пікірлер: 24
isn't count 500 X 500 = 250K ? which is 10 time more. Or are we assuming only 10% of friends will be mutual or something like that?
@jordanhasnolife5163
16 күн бұрын
Oof yeah good catch. Point is, fan-out probably won't work here.
Thanks for the great content again! Question: In your final diagram, the middle flow (new connection service) shows two layers of Kafka (a stateless consumer in between). Why do we need both layers? Can't the "new connection service(s)" directly push to the corresponding Kafka shard and avoid having the Kafka layer and the stateless consumer?
When adding mutual connections from the Flink nodes. How is it known that the new mutual connections are not already direct connections? e.g. For 10: 3, 4, 15 you are creating 3,15 and 4,15. What if 3,15 and/or 4,15 are direct connections? These connections could also be on a different Flink node/partition.
@jordanhasnolife5163
12 күн бұрын
Fair point - you can always just hit the database first here. We will have a connections table sharded by user Id so we know where to look.
@ishallwin24
12 күн бұрын
Same doubt
I have a question on brokers and message queue. Do i setup the broker on a server and then set the consumers on other servers? Lets say i have a mail server and i need to classify the emails and send them after classification to there right system. Where do i host the broker and the Ai classification model?
@jordanhasnolife5163
16 күн бұрын
I mean you can technically set them up wherever, but ideally different containers yeah
Awesome , great video🎉
Does profile update mean updating the latest job or education? If yes, why do we need to update the mutual connection DB for that?
@jordanhasnolife5163
6 күн бұрын
Yes - because the data is denormalized in our mutual connections database
Keep doing its help us out so much
You are great bro
hey, i have some questions if anyone can please help me :) 1) when jordan says shard the database by userID, it means shard it by the hash of the userID (for consistent hashing)? 2) sometimes i see the term partitioned by instead of sharded by, are those the same?
@jordanhasnolife5163
14 күн бұрын
1) yes 2) I think so, others seem to disagree
@lalasmith2137
14 күн бұрын
@@jordanhasnolife5163 thank you so much for taking the time to answer :) also, can't thank you enough for all the knowledge i gained since finding your channel
Thanks for making this video Jordan. I have two questions: a) You mention "Mutual Cache table", but it appears you are using SQL db for that. Does not cache mean keeping in memory? b) It is mentioned that we need very fast reads ("fast as humanly possible"), should it not engender use of mongodb or something liek that instead of SQL db?
@jordanhasnolife5163
16 күн бұрын
Cache doesn't inherently mean memory, it just means having the result of a computation easily accessible. Why are mongoreads faster thansql?
Hi Jordan, loving this video. A couple of quick questions: 1. For the adding a connection workflow, is it supposed to be real-time processing or batch? 2. Let's say B accepted A's invite to connect and A wants to view the change right after it, how can we ensure that? 3. Does it make sense if we put the mutual connection data in memory cache servers and have a graph db to store the raw connections so that we can rebuild the cache if any node fails? Any idea or discussion is appreciated. Thanks!
@jordanhasnolife5163
13 күн бұрын
1) Realtime 2) You could first write to a table before using CDC to sink to Kafka and then see the first degree connection there
Thank you :)!
W as always
How mutually awesome
Watched. --