in the executors section, i can see u have 2.2GIB available in the executor, but i see three columns on the right side (input, shuffle readm shuffle write) should i consider the sum of these 3 in order to check how much did i use of storage in total? or should i consider only the input column. This will help me to check how much memory can i allocate since the beggining.

@backstreetbrogrammer

3 ай бұрын

Thanks @garydiaz8886 for QnA. Total memory available for the executor is 2.2 GB and we can always see "Storage Memory" column as / . As mentioned in the video, the is showing as 0 because in our sample program - it was so less memory being used that it rounded off to 0 and the program is already completed and waiting for scanner input to terminate. As the UI is real-time, it will display the current memory being used which is 0. For the other three columns - its correct to say that the is the SUM of these three columns. Just to note here is that shuffle read/write memory is only used for shuffle operations like reduce, group by, etc. Hope it clarifies.

I am saving dataframe to disk using persist , but spark ui still shows 0.0 storage and storage tab is empty. Could you please suggest me something ?

@backstreetbrogrammer

28 күн бұрын

Hi @serenitytime1959, please advise the size of the dataframe. If the size is very small (in KB), it may be rounded off to 0.0 as the display is taking default as GB. Spark also tries to optimize the data storage as much as possible by storing in memory rather than disk (which makes it much faster than MapReduce). Only the data spilled over (could not fit in memory) is stored to disk.

You are just saying what is listed in UI..it will be good if you explain more what they are and their use case

@backstreetbrogrammer

8 ай бұрын

Hi @rkshbhgt - here I have just tried to explain what is there in the UI and its more of a "features-definition" summary. For the use cases, I will try to include some videos in the end with real example program to demonstrate the same.