00:00 Introduction 05:30 Code: Benchmarking different ways to write to a file 31:18 Code: Benchmarking writing to a file using streams naively (faster but not memory efficient) 38:05 What are streams in NodeJS? Explanation with analogies 46:36 Different types of streams (with examples) 53:10 How do writable streams work? 1:01:05 How do readable streams work? 1:04:16 Code: Using writable streams to make previous program more memory-efficient 1:41:06 Reading NodeJS docs about writable streams and recap 1:51:44 Code: Using readable and writable streams in conjunction 2:12:45 Code: Selectively write chunk of data using streams 2:21:32 Explanation of what's happening with numbers being split when chunking the data 2:38:20 Code: Implementing the solution that solves the splitting issue 2:54:36 Reading NodeJS docs about streams and recap 3:02:16 Code: Example of piping 3:25:38 Previous problem using piped streams 3:31:36 Reading docs on pipes 3:40:10 Why you shouldn't stream.pipe in production 3:44:35 Example with stream.pipeline 3:48:42 The stream.finished function for cleaning up 3:49:18 Discussion about readable/writable streams, duplexes and transformers 3:51:15 Implementing our own writable stream 4:34:00 Implementing our own readable stream 4:53:22 Duplex and transform streams 5:09:38 Creating an encryption and decryption application 5:48:35 Done with streams: Final notes, advice, stream-based NPM packages, etc.

@1732ashish

Жыл бұрын

thank you for this.

@ziyadseykhanov3967

Жыл бұрын

Thanks man

@InvincibleMan99

Ай бұрын

Thanks a lot my dear. Have a great life

I'm really glad that I found your channel randomly. By watching all of your previous nodejs videos I feel much more confident working with nodejs ❤

Simply the fact you backed yourself into a corner and then walked yourself and the viewer through a possible solution, instead of using a cookie-cutter example, places you above the rest. And this is leaving out the fact you're such a good pedagogue.

wow, a video last 6 hours, can't wait to enjoy it. Thank you so much

man you re a legend like you explained the hardest subject in node better than anyone I watched, going in depth on the underline implementation of streams thank you so much for the content

after watching this great tutorial i feel like i'm the best nodeJS developer in the world hhhhhh thank you so much really such GREAT GREAT GREAT Tutorial

It's rarely the time when we get to learn so much on one topic from one video. Thank you so much man. I am eagerly waiting for your next video.

Loved this series so far. Would also like to see a condensed crash course on the same topic.

OMG WOW.! I have been trying to understand streams for so long, I have never came across such a detailed video on streams till date. YOU ARE A GEM. I really really appreciate what you are doing and request you to keep posting more videos when you get time. Love from India.!

No words to describe this explanation. I already used streams, but having this understanding of how all works it's another level. Plus, if you are a full stack developer you can use this concepts in the front-end with web streams.

Man, mannnnn, maaannn!!!!! Seriously!! I have been looking for this for over a week and couldn't find thorough details about node stream. So much grateful for this 🙏🙏🙏🙏 Thank you

A video like this is an absolute TREASURE FOR LEARNING FUNDAMENTAL NODE CONCEPTS! Thank you very very much for creating helpful tutorials like this one. 😀

I thank you very much for this tutorial. Glad that I found you channel. Learned a lot from you. Hope you are doing good and please please don't stop this series, waiting to learn more from you

Thank you so much ! I have learned a lot from this video about streams. And the way you explained each and every concepts is AMAZING !

Hey you are doing great work there is no content like this on youtube... It'll help a lot if continue this level content....

Thank you for your detailed explanation on what happens internally. I couldn't wrap my head around streams by just looking at the sample codes online.

the most intensive nodejs stream course. thanks for sharing.

thank you for your tutorials!! please continue making such high quality videos! really a treasure!

Thank you for all your efforts explaining those concepts that seems hard to master at first glance. Your videos make the understanding of node core concepts much more easy and this is great quality content and plus, for free. You are thrully contributing to make the world better

it took me sometime, but i finished!! I really love your advice of looking into the source code of libraries and of nodejs itself. since I started your series I feel i can read the nodejs docs more confidently, before they were obscure to me lol. Erik Wendel has a bunch of videos with project with streams, but he explained superficially, for a slow learner like me, I was lost. Thanks to you i feel I understand streams now. I implemented the percentage thing of the encryption app. now I going to start your networking video and also watch the projects of erik wendel, this time, understanding what he is doing haha. thanks for the great content

thank you so much for the tutorial. This helps me understand streams a lot.

Thank you, your course is extremely helpful!

More than a nodeJS tutorial.You are the best

wow... it's there, can't wait to watch it, thank you.

Great videos, explained in a manner for all public.

You're the best bro, Thanks for this series.

Wawww.... Worth waiting for a long time 👏👏👏

Stream has always been i topic that interests me the most. This will be great for revising my knowledge and learning smt new along the way!!

Wow...kindly dnt make us wait for these type of awesome videos...kindly keep loading❤️❤️

Wow, another amazing video. Many thanks❤

Another great video! Thank you!

awesome explaination. GREAT WORK! Thank you :)

Thanks for the new year present!

i'm really glad that see this video about nodejs stream

man i am excited to begin this journey, already watched all the previous videos, let's gooo

@Cododev

Жыл бұрын

Keep it up buddy! Glad you’re learning so much!

WOW! 6 hours on streams! amazing stuff, well explained. keep up the good work

@Cododev

Жыл бұрын

Thanks, will do!

This video is pure gold!

you are awesome I never see any tutorail on internet using approch for learning like you use. Just go deeper and deeper.

👍 Really helpful please continue making this type of content

This is a long one. I am 3hr in. Very good content!

You made a 6 hours+ video on stream!!! Thank you.

Amazing course understanding node js fundamentals and going in depth is what i like about your courses unlike other node js courses they just into using Express js and make everything unclear and abstracted thanks for your efforts it really changed my pov in node and backend fundamentals and if anyone is wondering ,I highly recommand this course watch plus code along with him and take notes ,Cant wait for you next one .

@Cododev

7 ай бұрын

Much appreciated!

Very important topic, thx!

You deserve to have a million subscriber lucky to find this video

It's a very important topic of CS.Great great great thank to you.

Hats off brother. God bless you

Really thanks for your video~❤

thank you very mush for all the information you shared with us ,it really helpful 🙏🙏❤

The emphasize to use pipeline() instead of pipe() is very important.

man great content, congratulations!

WOOOOOOOOOOOW , thx for your efforts and time ♥♥

a minor bug at 1:29:25 - breaking the loop before incrementing the counter. It will result in some duplicate numbers - 2915, 5645, etc. The counter should be incremented before breaking the loop

@Cododev

10 ай бұрын

Thank you very much for mentioning this!

Wow i just found a hidden diamond, im gonna watch all of this !!

Great stuff!

on 5:30:13, the 92 which is 5c in hexa is the result from the operation: 1116 % 256 = 92

Your voice is truly expressive and wonderful. It's very gentle and easy to absorb. A slight improvement would be to create engaging slides before diving into code demonstrations. That would be even better

@Cododev

7 ай бұрын

Thanks Emma! I'll keep that in mind!

THANK YOU!

2:24:00 If im not mistaken, there is a possibility that last number in file can be split into two separate chunks, so it is not certain that last item in last array is 'expected string number'?

One more Nodejs Core Concept ❤

Thanks so much for providing in depth explanation. ❤ 3:32:54 Is it good to use timeEnd in writeStream.on("finish") instead of readStream.on("end") ?

Thank you bro❤

You deserve way more subscribers

would love to see a video explaining how express builds upon node

Great video!

OMG. What a u doing bro. this is just perfect. Greetings from Moscow

Hi, a very good video. can you tell me the font name in your code editor? It looks pretty.

Bro, you're the best 👌

You're awesome bro

wow! amazing.

thank you so much

Great job! Just bought your udemy course, keep it up! I´m wondering wht´s the name of the vscode theme in the video?

The best of the best ♥

this is epic i like your teaching style by giving examples and optimizing as max as possible from where you learnt this much 😁

@Cododev

Жыл бұрын

Aww thanks! Well just experience and a good deal amount of research and experimentation!

During the "Implementing our own writable stream" section at 3:51:15, can some one verify that when subclassing the Writable class, the Writable class provides us an internal buffer and also manages it for us too. And that we don't actually have to implement/manage our own internal buffer in the subclass like we did there with the chunks array. tysm.

Will you do an async video? (promises, callback, Promise class..., How it works...)

You are My God in human shape.

What is the name of the background music?

Hi , @ 04:57:40 you suggest that in case of transform stream , data chunks move from readable internal buffer to writable internal buffer. I think it should be the other way around , we write chunk to the writable internal buffer which then goes through the transform function where we push the transformed chunk to the readable internal buffer from where it can be consumed.

@Cododev

Жыл бұрын

That is an excellent point, thanks a lot for mentioning it! I’ll soon do an update and fix this, thanks again!

@niktwister

Жыл бұрын

@@Cododev All thanks to you man for your efforts. Great videos 👍

Could you please create more videos in nodejs

Please i can't find the video about buffers

Where can i found more information? is there a roadmap?

Thanks!

@Cododev

Жыл бұрын

Thanks Matt!

Hi I have a doubt. Please look at the below code. const Stream = require("stream"); const readableStream = new Stream.Readable({ highWaterMark: 200, read() {}, }); readableStream.push("H"); readableStream.on("data", (chunk) => { console.log(chunk.toString()); }); In above code, I mentioned the internal buffer size as 200. I pushed only the letter H onto the stream. According to the tutorial at 1:02:14, data event is raised only when internal buffer is filled. But the above code is printing the letter H(meaning - data event is raised before buffer is filled). Could someone explain me please?

5:07:30 If your duplex.write is not writing to the file, make sure to add 'duplex.end(Buffer.from...) for the last line, to trigger the _final method. Happy coding

I'm not finding the course on nodejs process

I am Windows user, with ryzen 5600U. When i tried to run the "splitting chunk 2:38:00 " code with 10^9 size file, memory and Cpu work as it is, but disk r/w become too low, may be around 10-20MB. But in your code whenever you execute that code, your disk r/w go up to 1GB. One more thing is, if i perform stream write() operation with chunk, outside the foreach loop, disk r/w become high up to 800MB. Please help, me how to increase this disk w/r speed. This 10^9 file took at least 39minit to add all the even number in "dest.txt" file in my system.

Please keep making videos

I'm encountering an issue with a write-many.js. When I write a series of simple indexes ranging from 0 to a set number of writes, I consistently find two instances of the number 103978. I checked every number while writing even numbers from a file named src.txt, and discovered that the source of the problem is within `src.txt’ itself, which contains two identical numbers (103978). What could be causing this duplication?

great stuff, a question though @01:22:04 , if we allocate internal stream lets say 16383 bytes (writablelength = 16383) and then we allocate same stream a single byte but with setTimeout , the writableLenght is just one byte. for example: stream.write(Buffer.alloc(16383, "a")); console.log(stream.writableLength); // 16383 setTimeout(function () { stream.write(Buffer.alloc(1, "b")); console.log(stream.writableLength); // 1 ---why? }, 1000);

@Cododev

Жыл бұрын

Well, it could be because this example is not really considered an example of repeated writes. We are writing once to the file (assuming it is a file stream), and then there's some time for the stream to flush the data to the underlying resource because of that setTimeout. So after the first write, Node sees the setTimeout, and then the main execution thread is done. And after a little while, the main execution thread will kick in again and handle the second write (learn about the event loop please if this doesn't make sense). I'm not sure but I think when we are going to the next tick (stopping the execution thread and then starting it again), Node streams will do an automatic flushing and draining so that the write is now done and the internal buffer is empty again. Let me look more into this and I'll let you know if I find more info about it! Thanks for the question!

@RaGa_BABA

Жыл бұрын

@@Cododev yeahhh..it makes sense.. thankyou for the detailed answer.

UTF-8 decoding a chunk only works if the characters are ascii characters ( 1 character = 1 byte ), anything else (example Chinese characters) uses more than 1 byte, so if you try to decode the chuck it can either throw an error or give you incorrect characters

@Cododev

Жыл бұрын

That’s an excellent point! I’ll soon add a description in the pinned comment and point this out. I should’ve said every ASCII character is 1 byte, not every utf-8 character. Thanks a lot for mentioning this!

@soniablanche5672

Жыл бұрын

@@Cododev I've even noticed the Node documentation do this in their http request stream example where they immediately decode the chunks as utf8

@nsudhir_here

Жыл бұрын

Hii, It seems you're understanding how numbers are being split. I need a little bit to understand it. Can you help?

Thank you for making this available to all for free. Your content is very high quality - far better than the Node.js content on paid platforms like Pluralsight. Do you have a link to donate directly without giving 30% to KZread?

@Cododev

7 ай бұрын

Thank you very much Joseph! I don't have a link for donations, but if you want, you could purchase the new version of the course using my link from the latest KZread video. Udemy will give 97% of the revenue to the instructor if a purchase is made using the link. Otherwise, the instructor only gets 37%, so that would be a great support! Thanks again!

@Joseph-ex4sf

7 ай бұрын

@@Cododev Just purchased it. It's the least I can do. I've learned a lot already! Very few instructors go as deep as you do on the material. You rock man. Looking forward to seeing what comes next.

Hello..got one more question from @3:21:08..(I hope you create a discord server or something😅) when I copy using below code it doesnt produce any null/weird characters in the des.txt file and it's faster as well..why?? let buffer = Buffer.alloc(16384); let stats = await readhandle.stat(); let itrs = stats.size / 16384; for (let i = 0; i let data = await readhandle.read(buffer, 0, 16384, null); writehandle.write(data.buffer); }

why on 4:32:16, number of drains is 480 while number of writes is 481? This hasn't account for the last write which flushed the remaining data

Just commenting to get you more reach, upload more content, thank you

🥳🥳🥳

At 24:33 can anyone explain why it’s taking 700mb of memory?

4:32:36 How come you didn't have to call stream.end when you used your own implementation of a writable stream? also why is there +2 more writes than drains? doesn't it call drain every time you write?

Hey there! I was wondering if you could consider including your course in the Udemy subscription plan. It would be fantastic to have access to your content through such a convenient platform. Thanks for considering it!

@Cododev

2 ай бұрын

Yes I’m looking forward to doing that, hopefully it will happen sometime soon if Udemy doesn’t require me to remove all the Node.js videos from the channel.

do you have a discord group channel?

Hi question for readStream @2:40:00 , for the split issue. We are able to solve the issue where first number and last number of chunk are getting messed when we know the chunk array (all elements of chunk are in increasing order) but if we dont know the content of readStream chunk , how can we resolve this issue.

@Cododev

Жыл бұрын

Hi there! Well that's why it's important to understand the underlying format of the file you're dealing with! So suppose instead of a text file, it was an image file, then you would need to know how the pixels and colours are represented in binary so that you could do some modifications like applying filters, cropping, resizing, etc. In this example we made use of the fact that numbers are in ascending order, but honestly, for each problem you will need a specific solution that works best for that. So if you could maybe narrow your question down a bit, I might be able to explain better what to do in that specific situation. Thanks!

@gabrielmachado5391

Жыл бұрын

a chunk just support 65536 bytes. if it has just 2 bytes lefting to full it, and the last string have 5 bytes, then the string will be splitted, two bytes would left behing to the previous chunk and the another three bytes will to the next chunk. Then, to solve you problem, you can see the full chunk to see the content of the chunk and then do some action based on what you need. see the full chunk content logging: console.log('------------staring chunk------------') console.log(chunk.toString()) console.log('------------ending chunk------------')

@RaGa_BABA

Жыл бұрын

@@Cododev Hi thanks for the reply.. I'm not a developer or programmer..its just a hobby so I might not be able to frame question very accurately. Suppose I'm reading a large mkv file from server how can I write it into another file without messing the data.is it possible without using pipe.Am I suppose to know the mkv formate One more question other than this😅. In readstream the default highwaterMark is 64 KiB and for the writeStream its 16KiB..my question is when data event is fired the Read stream has 64KiB data but write Stream can only take upto 16Kib before it needs to drain so does the 48KiB data rest into memory or how does it play out..does write stream writes into destination file in 3 chunks(16* 3 = 48) or all the 48KiB data gets written at once before reading another chunk.

@RaGa_BABA

Жыл бұрын

@@gabrielmachado5391 thanks for the input gabriel..but if we don't know the underlying format how can we spot where the error has occurred that we need to fix by adding the last byte of previous chunk with the first byte of next chunk

@gabrielmachado5391

Жыл бұрын

@@RaGa_BABA in node docs, they explain that chunk could be either a buffer or a string. otherwise, to catch errors you can use the 'error' event. descripition of this event, according the docs: "The 'error' event may be emitted by a Readable implementation at any time. Typically, this may occur if the underlying stream is unable to generate data due to an underlying internal failure, or when a stream implementation attempts to push an invalid chunk of data. The listener callback will be passed a single Error object."