Are bhai ye admi abhi tak viral kaise nahi hua he deserve it

Thank you Manish Bhai, very helpful

Useful content, Much thanks Manish bhai :)

Thank you Manish Bhai for this gem

Very helpful video.. ek interview question tha .. how to validate schema and null values.. please ek real time scenario as eg le kar bata do

thanks for great quality content

Thank you Guru!

Very helpful

Thanks Manish!

Good work upload something on streaming

ThANK U GURUGHEE...

thanks for quality content.very helpful for fresher...

Yar manish kuch ispe v video banao Cv p kya project dale Aur Different technology se Data me switch karna hai to kya resume pe hona chaiye

Please add a video on how to handle null value and how to validate a scheme

Thanks very much for the tutorial :) , I have a query regarding reading in json files. so i have an array of structs where each struct has a different structure/schema. And based on a certain property value of struct I apply filter to get that nested struct , however when I display using printschema it contains fields that do not belong to that object but are somehow being associated with the object from the schema of other structs , how can i possibly fix this issue ?

Thanks. How can be extend this in case we have a stringified json and we need json fields inside that .

Bhai, there are much simpler functions like json_tuple to extract columns from nested json inspite of exploding columns.

@manish_kumar_1

11 ай бұрын

Oh I didn't know. Let me check that.

RDD k practical ya theory quetions aate hai kya interview me?

can we define our own schema using this json

is nested data and complex data both are same?

Hello Manish , Just one doubt , what if we have 100 columns of struct type and 100 columns of 100 type. Should we write explode and .column 200 times. Or is there any other way please help to this find out. Thanks

mere data me leveling nahi he, lekin ek column me data ke andar list k andar list he usme schema me bhi string dikha raha he or leveling nahi dikha raha, any idea?

.select use karke we are selecting to expload can we use withColumn also ?

@manish_kumar_1

11 ай бұрын

Yes it will work using withColumn too

How many lectures remain for complete this series? Please start SQL playlist also.

@manish_kumar_1

11 ай бұрын

After 8-10 videos new playlist will come

i am learning python but when i go to geeks for geeks to solve easy question i cant be able to solve them like runner up questions,or etc , can you guide me regarding this

@manish_kumar_1

11 ай бұрын

Starting me tough lagta hai. But slowly aapko samjhne lagaega, pattern to solve questions.

how to make a generic json parser(flatnner) function, do u have that code , could you please share , here u have hard coded, but any generic funciton

@manish_kumar_1

11 ай бұрын

I have not written yet. I will try to write a generic function that will flatten the entire json into dataframe

@ETLMasters

11 ай бұрын

I think you are talking about this: from pyspark.sql.types import * from pyspark.sql.functions import col, posexplode_outer def flattenDataFrame(explodeDF): DFSchema = explodeDF.schema fields = DFSchema.fields fieldNames = DFSchema.fieldNames() fieldLength = len(fieldNames) for i in range(fieldLength): field = fields[i] fieldName = field.name fieldDataType = field.dataType if isinstance(fieldDataType, ArrayType): fieldNameExcludingArray = list(filter(lambda colName: colName != fieldName, fieldNames)) fieldNamesAndExplode = fieldNameExcludingArray + ["posexplode_outer({0}) as ({1}, {2})".format(fieldName, fieldName+"_pos", fieldName)] arrayDF = explodeDF.selectExpr(*fieldNamesAndExplode) return flattenDataFrame(arrayDF) elif isinstance(fieldDataType, StructType): childFieldNames = fieldDataType.names structFieldNames = list(map(lambda childname: fieldName +"."+childname, childFieldNames)) newFieldNames = list(filter(lambda colName: colName != fieldName, fieldNames)) + structFieldNames renamedCols = map(lambda x: x.replace(".", "_"), newFieldNames) zipAliasColNames = zip(newFieldNames, renamedCols) aliasColNames = map(lambda y: col(y[0]).alias(y[1]), zipAliasColNames) structDF = explodeDF.select(*aliasColNames) return flattenDataFrame(structDF) return explodeDF

Hi Sir, How many videos still left to complete.

@manish_kumar_1

11 ай бұрын

A lot of things to learn but after 8-10 videos we will move forward with others topics.

data = [("openai",'[{"name":"ram","work":"salesman"}]'), ("tech support",'[{"name":"lakhan","work":"service man","lname":"mishra"}]'), ("data operator ",'[{"name":"lakhan","work":"service man","salary":"5000","System":"del"}]')] Bhaiya ji Jo data upar diya h ise kese flatten karenge ye mere client project ka sample data h please help me

Manish bhai I see all your videos of this play list but mujhe meri problem kas answer nhi mila me nested json dataset pr work kr rha hu Wo load to ho rha hai but show me corrupt record arha hai mene uska schema type change kia to jo dataset hai Wase he show horha hai json me me Ise kase load kru mene multiline bhi use kia

@manish_kumar_1

9 ай бұрын

Aap mujhe may be a smaller data set mail karo in a file or linked par bhejo

@DedloxGMR

9 ай бұрын

@@manish_kumar_1 bhai linked in pr nhi horha aap mail bata dijiye

Hello manish, Ciuld ypu provide the json file which you have used here?

@manish_kumar_1

3 ай бұрын

Data download karne ka link description me hai

@dineshughade6741

3 ай бұрын

Oh Great, thanks Manish

you havn't started this vedio with `Possible Interview Questions`😅

@manish_kumar_1

10 ай бұрын

Code hi likhwayenge isme