•\tData and AI Architect
•\tend-to-end serverless and data solutions over cloud covering a range of cloud services including AWS Big Data Stack, Data Lakehouse, Apache Spark, MongoDB, ETL, Azure Big Data Stack including ADLS, Azure Databricks
•\tLed and developed Generative AI based discussions and proof of concepts covering various use cases like text summarization, sentiment analysis, Q&A
•\tHelp clients understand the significance of modern data architecture; the ability to drive the value from the data in a cost effective and efficient way
•\tTeam Leadership - enabling teams on new and emerging technologies
neeraj garg
#neerajgarg #generativeai #coumptervision #naturallanguageprocessing #nlp #largelanguagemodels #deeplearning #machinelearning #awstutorial #dynamodb #mongodbtutorialforbeginners #mongodbtutorial #mongodb #bigdata #apachespark
#cloudcomputing
Пікірлер
Hi @NeerajGarg, I really loved this video, but I WOULD LIKE TO KNOW LIKE SUPPOSE I HAVE 3 EC2 INSTANCES RUNNING IN DIFFERENT IP'S. 13.201.228.176 65.0.179.114 13.232.181.75 HOW CAN I CREATE THE SHARDED CLUSTER ON IT. AS I HAVE ALREADY INSTALLED MONGODB INSIDE IT SUCCESSFULLY. PLEASE GUIDE. BY THE WAY, LOVED THE CONTENT ❤
I will embeed publisher id in each book
Thank you so much, man. I was struggling so much with how to set up sharding with MongoDB even when I tried to follow the official documentation. This was the only video that I found that actually helped me and now it's working perfectly! Thanks again :)
Thank you so much for sharing your views and nice gesture. I'm glad that this has helped you in some way.
any downside, latency if add members using public IP sir?
Hi, thank you for sharing your query. Can you please elaborate your use case and the requirement. That will help me to suggest in an appropriate way.
Hi, I am beginner to mongodb, while learning through videos , at the end of every video, i cant see text as the next suggestion videos of your series blocks the content. Its very disturbing. kindly remove it. Will be helpful.
Appreciate you taking time to share the feedback. Thank you.
Help me sir
Please watch video and have patience
Hi sir, thank you for the video. Very useful. I have a quick question as I'm very new to MongoDB. According to your embedding approach example, user with access, what if one day we need to query only the access information regardless of the users eg. how many access are there currently ... ? Should we change it to a referencing approach and a user will have a reference to access's object id instead? Or we can still use embedding in that scenario?
Hi, thank you for sharing your feedback and query. There could be couple of options. You can aggregate the access information and put in some collection based on time etc. (if that is the use case). OR there could be another option based on the access pattern. You should list all the possible query access patterns on your data. Then you will be able to find out the appropriate approach of selecting embedding or referencing. Hope this helps. please don't hesitate to reach out if still in doubt.
Thanks for this series sir 👌
👍👍
Great learned something new.
Thanks a lot Amit Bhai, I'm glad that you got something
after creating pipeline, how do you run it?
Hi, The aggregation pipeline in compass given an option to export it to various language constructs like node.js, python or shell. The given code can be embedded in your choice of query execution engine ( programming language or shell). db.collection.aggregate(pipeline, options) an example from mongodb docs: db.orders.aggregate( [ { $match: { status: "A" } }, { $group: { _id: "$cust_id", total: { $sum: "$amount" } } }, { $sort: { total: -1 } } ] ) The following links can also be looked at for more details. www.mongodb.com/docs/manual/core/aggregation-pipeline/ www.mongodb.com/docs/compass/current/create-agg-pipeline/ www.mongodb.com/docs/manual/core/aggregation-pipeline/#run-an-aggregation-pipeline Hope this helps you. Please don't hesitate to get back.
Good
sir, I have a doubt. I am new to backend development. My question is, whether this serverless mongodb plan will handle user volume of 2000 at a single time with the read/write operations from my mobile app as frontend.
Hi, thank you for sharing your query. Can you please help me understand more about your use case. What could be the potential maximum peak load at any time. Does the load (2000 users at a time) mentioned by you hold true for whole day or any specific time of the day. Is it true for all the days or it is for a certain time of the day/week/month/year. How have you arrived to a figure of 2000 users at a single time. Is this figure the maximum or you are expecting more load in near future or in the long term. Having explored all the above queries ( there could be many more based on your answers), I would recommend to do a proof of concept with serverless and then decide if your requirements are fulfilled by serverless (assuming you have complete insights around your workload, the size of collections and data, the shards, the indexes and the query performance). Please let me know if this helps you and feel free to post your queries. You may also like to explore the cost of serverless vs shared/dedicated servers www.mongodb.com/pricing Best wishes!!
👍
Sir Mara account kholo😢
Please watch the steps in the video
👌👍
👌👍 very informative
Glad you think so!
Paano po mag balek nang account
Can you please translate your question in English.
hi, i also got the same error as other in mongos instance - You are not connected to a mongos. This command may not work as expected - MongoServerError[CommandNotFound]: no such command: 'addShard'" i guess nohup mongos --configdb ... is not working i checked after running that and there is no mongos running in my instance note: for this instance, i also installed full mongo-org, just like the others please help me. thanks
Please share more details like the OS version, MongoDB version Are you experimenting on a single machine or it is a cluster of nodes. is there any issue in networking or VPC etc. Thank you for reaching out with your query.
Great Explanation sir I have one question sir if I have around 27 gb of data in mongodb database and we have some huge collections which contains 80lacs data for all different projects and users are facing buffering issues, loading issues so is it good if we go with the replica set approach or we can create separate collection for different projects. Our server has configuration of 64 gb of RAM and 6cores. Please suggest some approach or solutions to resolve this issue.
Thank you for your kind words. In order to help you, need more details as mentioned below: Have you considered using indexes. If not, please create the indexes on the slow collection. If these indexes have been created, try using explain to understand the query which is running slow. kzread.info/dash/bejne/apiYl5ShXbCdqJs.html kzread.info/dash/bejne/hpp-yaSbhMmXgrQ.html Have you considered sharding your collection. If not, please select the appropriate shard key and then observe for a certain timeframe. This should help as well. kzread.info/dash/bejne/rHt6wZmkgaWyoZc.html Check my videos on schema design best practices and see if there are any issues related to schema design. kzread.info/dash/bejne/jJmI2dqOZKaWZ9I.html kzread.info/dash/bejne/no542q-jeNfHf7w.html All the above points are so critical and can help you to get rid of performance issues. If these don't help, please feel free to share your concern again with more details like sample documents, kind of queries you have been running, indexes created, output of explain command. Hope this helps. Best wishes !!
@@NeerajGarg Thanks alot sir for helping me out.
👍keep it up
Thank you for encouraging words.
Nice start
Please move your picture to the top when you are explaining something in the lower part of the screen. A smaller pic would be more appreciated :)
Thank you for sharing your feedback. Point notes and will be taken care of in future videos. Thank you once again.
Hi, was wondering if it’s possible to create a docker-compose.yml file that could create dynamodb locally with a table and items in it. I am working on it and find that the container is unable to follow the AWS commands such as aws dynamodb create table inside the running container (throwing command not found error). Also tried using .sh file …. Still no luck.
To run aws commands , aws CLI is required followed by the configuration. I need to check this as well. You have raised a good point. Thanks for sharing this.
You should also explain why it is behaving this way
Hi, please see if you can elaborate your feedback a little. I can try to address it. Thank you for taking out to share your feedback.
👌👍
Hello sir i am not able to connect this command nohup mongos --configdb config_repl/localhost:28041,localhost:28042,localhost:28043 --bind_ip localhost & receive this error : [1]+ Exit 14 nohup mongos --configdb config_repl/localhost:28041,localhost:28042,localhost:28043 --bind_ip localhost
nohup in linux is used to run the process in the background. this is the expected behavior of the command. Can you try running ps command to check if mongos process is running in the background. Also, will request more details like OS version, Mongo version etc. Hope, you have followed all the steps listed in the video.
Hello Sir, Could you please let me know the entire process for do the mongodb database point in time restore using the mongodb opps manager console?
Thank you so much for your query. In case you haven't found any solution around this, I can check and confirm. Please let me know.
@@NeerajGarg Please do sir. Because I am not able to find any documents regarding this. Hence requesting you to please share if you have any documents with you please.
Such a good channel but idk why it is so underrated.. there should be more viewers and subscribers.
thanks a lot for your feedback and comments
While configuring the config server i am receiving the below errors in never ending loop on ubuntu Linux command: mongod --configsvr --port 27020 --replSet rs0 --dbpath /u/mongo_date/configdb01 --bind_ip localhost Exceptions: error","attr":{"error":{"code":26,"codeName":"NamespaceNotFound","errmsg":"Unable to retrieve storageStats in $collStats stage :: caused by :: Collection [local.oplog.rs] not found."},"stats":{},"cmd":{"aggregate":"oplog.rs","cursor":{},"pipeline":[{"$collStats":{"storageStats":{"waitForLock":false,"numericOnly":true}}}],"$db":"local"}}} {"t":{"$date":"2024-01-15T15:04:07.002+00:00"},"s":"W", "c":"QUERY", "id":23799, "ctx":"ftdc","msg":"Aggregate command executor error","attr":{"error":{"code":26,"codeName":"NamespaceNotFound","errmsg":"Unable to retrieve storageStats in $collStats stage :: caused by :: Collection [local.oplog.rs] not found."},"stats":{},"cmd":{"aggregate":"oplog.rs","cursor":{},"pipeline":[{"$collStats":{"storageStats":{"waitForLock":false,"numericOnly":true}}}],"$db":"local"}}} {"t":{"$date":"2024-01-15T15:04:07.004+00:00"},"s":"W", "c":"QUERY", "id":23799, "ctx":"ftdc","msg":"Aggregate command executor error","attr":{"error":{"code":26,"codeName":"NamespaceNotFound","errmsg":"Unable to retrieve storageStats in $collStats stage :: caused by :: Collection [local.oplog.rs] not found."},"stats":{},"cmd":{"aggregate":"oplog.rs","cursor":{},"pipeline":[{"$collStats":{"storageStats":{"waitForLock":false,"numericOnly":true}}}],"$db":"local"}}} {"t":{"$date":"2024-01-15T15:04:07.593+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read concern majority reads are currently not possible.","nextWakeupMillis":12200}} {"t":{"$date":"2024-01-15T15:04:08.000+00:00"},"s":"W", "c":"QUERY", "id":23799, "ctx":"ftdc","msg":"Aggregate command executor
I'm sorry to hear about the issue. However, can you please share more details like OS version, MongoDB version. Are you running it on a single node or over multiple nodes. Also, please see if you can share why you didn't select MongoDB Atlas (managed service). thanks. Look forward to hear from you.
Any clue why config/shard server keep going down frequently? where to look for logs, config files etc? also i am running each shard and config server on 3 different ec2 instances and mongo Router on 4th ec2 instance?
Hi Mayank, apology for the delayed response. Please see if following helps. Also, please share more details about your env i.e. which MongoDB, OS version are you using. Why didn't you choose to use MongoDB Atlas (managed service). Logs can be configured www.mongodb.com/docs/manual/reference/log-messages/#configuring-log-message-destinations www.mongodb.com/docs/manual/reference/command/getLog/#getlog Please let me know if this helps. Would be glad to support in whatever capacity I can. thank you for sharing your issue.
Congratulations neeraj
how to consult
What is this regarding. You can send me a message on LinkedIn as well.
@@NeerajGarggot my first client, building a blood bank mern project currently using mongo atlas free one, how to proceed and deploy when project finishes which atlas plan should i buy and hosting too
Many congratulations on your first client. Best wishes for successful delivery. The product deployment and selection of plan will depend on a lot of factors like the expected workload (number of users, what time, how many requests per second/ per day etc. to name a few) on the DB. You can connect me on LinkedIn and we can have more detailed discussion over a call, if required. This platform may not be appropriate for you to share all such information. Hope, this helps.
@@NeerajGarg thanks a lot! what's you linkedin and happy new year 🎊
www.linkedin.com/in/neerajgarg5/
Hi, where to get database files which you used "zips.json". can you share that too. TIA!
Please share your email id. Will share over there. I will have to find those as my system was attacked by ransomware
Hi , sir I have a problem : the collection test in chunkMetadata have only 'shard_repl1' not like you the two shards .how to do?
Hi, apology for late response. Can you please share the MongoDB version and operating system you have been using.
Sir where are the sample data files zips and test (json files from which you imported data)?
I think I've forgotten to share the link here. Let me find out the underlying data and share with you. Can you please share your email id with me.
True True False False
Thank you so much for reverting this. appreciate !!
It's very helpful but after adding these lines "mongosh --host localhost --port 27017" and "sh.addShard( "shard_repl/localhost:28081,localhost:28082,localhost:28083")" console shows this error "Warning: MongoshWarning: [SHAPI-10003] You are not connected to a mongos. This command may not work as expected. MongoServerError: no such command: 'addShard' " Please help in this out
Hi Dinesh, I'm really sorry for a delayed response. Can you please share MongoDB version, OS version etc.
@@NeerajGarg Sir i am having same problem , I am using windows OS and version of MongoDB is 7.0 , I tried to shift the version from 7.0 to 4.2.25 but the error remain same and i am not able to solve this problem
hello @pranav. thank you so much for reaching out with your issue. can you please share more details around the issue you have been facing. are you using windows 10 or11. Are you trying to setup sharing on a single node. you might have missed some step. Can you please share the URL of the steps you have been referring to. Will have a look and try to suggest some alternative.
Great tutorial Sir but in dynamodb, mongodb like can we have the aggregate option
thank you for your encouraging words. There is no aggregate option in DynamoDB. MongoDB is much more advanced when it comes to aggregation. One option in DynamoDB is to use aggregated values in the DynamoDB.
Please give the collections folder
Hi, are you looking for github.com/neerajg5/mongodb-tutorial/blob/main/mongo-sharding-git.txt OR please elaborate your requirement
Good sir Please keep it
thank you so much for your kind words
Hello sir
C:\Upskilling\dynamodb_local_latest>aws dynamodb list-tables --endpoint-url localhost:8000 'aws' is not recognized as an internal or external command, operable program or batch file. C:\Upskilling\dynamodb_local_latest> when I typed this showing aws not recognized after following ur step ..missed ? For doing all this do I need to get a aws account (not paid nor given any card details) ?
Amazing Tutorial Sir. Can you make some videos of redshift also? Thanks again.
Sure, will try. Occupied deeply in some personal engagements.
aws dynamodb list-tables --endpoint-url localhost:8000 'aws' is not recognized as an internal or external command, operable program or batch file. i am getting this
Sir, your teaching method is excellent. Do you offer any recorded sessions or training specifically on MongoDB DBA, especially focusing on backup and point-in-time recovery?
Hi, thank you so much for your encouraging words. As of now, there are no recorded videos other than youtube. However, I can organize a training if you can share more details around your requirements and the use case you are trying to solve.
@@NeerajGarg thanks a lot for your reply, Could you please create a tutorial on Point-In-Time Recovery (PITR) in MongoDB? It's a crucial topic for many of us and your practical demonstrations make complex topics easy to grasp. Thank you in advance
I think the videos are not in order. Please arrange the videos in playlist
sure, thanks for sharing.
Hey Neeraj, great content! Thanks for sharing.
Thank you so much for sharing your feedback.
Hi Thanks for the video Can you please the excel sheet url. It's missing in description
Please check this link github.com/neerajg5/mongodb-tutorial/blob/main/mongodb-cost-comparison.xlsx
Hi - I am facing the same issue which others have been facing. I have created the same, replica set created but I am unable to connect directly to mongos using mongosh -host localhost -port 27017 . In place I am connected to mongo. i.e. to > test Not [direct: mongos] test>. Hence unable to perform any sh command like sh.addShard If i use any other port like : mongos --configdb config_repl/localhost:28041,localhost:28042,localhost:28043 --bind_ip localhost --port 27517 [direct: mongos] test> does come up. However on trying to add the shards, gives the below error message: Could not find host matching read preference { mode: "primary" } for set shard_repl. Status: { apiVersions: [ 1, Infinity ], returnsPromise: true, serverVersions: [ '0.0.0', '999.999.999' ], topologies: [ 'ReplSet', 'Sharded', 'LoadBalanced', 'Standalone' ], returnType: { type: 'unknown', attributes: {} }, deprecated: false, platforms: [ 'Compass', 'Browser', 'CLI' ], isDirectShellCommand: false, acceptsRawInput: false, shellCommandCompleter: undefined, help: [Function (anonymous)] Help }
Hi can you share the MongoDB version you are using along with operating system details. Let me try replicating the issue at my end. Also, it would be helpful if you can send me a txt file with steps you have followed. One more question: why have you used --port 27517 in the mongos command. Did you try running without this port.