Excellent! Thank you for making this helpful lecture! You relieved my headache and I did not give up.

@ampcode

Жыл бұрын

Thank you!

Thanks for the video! It helps a lot

@ampcode

6 ай бұрын

Thank you so much! Subscribe for more content 😊

thanks sir, by far its working

You really helped me out a lot! Thank you! Followed you on LI.

Good explanation bro thank you 😊

@ampcode

6 ай бұрын

Thank you so much! Subscribe for more content 😊

you make good videos in easy understandable way. please make more videos on apache spark. like Spark architecture RDDs in Spark Working with Spark Dataframes Understand Spark Execution Broadcast and Accumulators Spark SQL DStreams Stateless vs. Stateful transformations Checkpointing Structured Streaming

@ampcode

Жыл бұрын

Sorry for late response. Yes sure I’ll definitely work on that. Thank you!

Hi! I did the same code and everything is running, but when I try to do a print(df) no data is retrieving. Do you know why it is happening?

if I follow same process for vs code then will it work?

In video #2, you already guided how to set up Spark on windows PC. And here again, you told to install using pip. Are both action required?

Can we use Google Colab instead of Anaconda?

But why installing Spark in the machine if we are going to install pyspark separately in Anaconda? Doesn't that mean we will only use the pyspark library frkm Anaconda? Someone to answer please

@ampcode

Жыл бұрын

Yes you are absolutely correct, I have made videos for both ways so that people can choose any of the installation method. Thank you!

KUDOS! I never found this type of APACHE PYSPARK series which includes real project-type scenarios.

Friend, when trying to create the session, i have the error: RuntimeError: Java gateway process exited before sending its port number I did everything right in your spark installation video, and pyspark works perfectly in CMD. The java is correct and everything works, what can it be? Help me please :/

@AnkitSharma9293

9 ай бұрын

Hi there, were you able to resolve this error?

Hi brother, since python will be installed automatically with Anaconda, will it be conflict with the python that I installed before? Thanks

@chesswithmoiz

Жыл бұрын

no bro

@ampcode

Жыл бұрын

Sorry for late response. No it will not conflict in the process and you can select the same in the project interpreter setting. Please let me know if any issues.

hey it is just given me file spyder is not ruuning

does anyone have setup in mac? please help me

I can't find spark 2.7 version.what to do with winutils version compatibility with the both

@epictales625

2 ай бұрын

the winutulis is compatible with the spark available

Hi Aashay, I am getting the below error while running the script through spyder/jupyter notebooks. Could you please help? Error- Py4JError: An error occurred while calling o29.legacyInferArrayTypeFromFirstElement. Trace: py4j.Py4JException: Method legacyInferArrayTypeFromFirstElement([]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.base/java.lang.Thread.run(Thread.java:1623) Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/04/30 17:55:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

@nikkzm3689

Жыл бұрын

Same here also bro :((

@ampcode

Жыл бұрын

Sorry for late reply. Could you please let me know if you have JDK, Python installed on your PC and environment variables perfectly set. If yes, we can discuss around this to solve your issue. Please let me know.

@safeats_

Жыл бұрын

@@ampcode same here. The environment variable are correctly set

@Ravenz13oomer

Жыл бұрын

@@safeats_did you solve it, I have the same error and all the environment variables are set

@safeats_

Жыл бұрын

@@Ravenz13oomer no couldn't set it

Why I get error , No Module named pyspark ?

@riyazkhanpatan4602

3 ай бұрын

You have to run "pip install pyspark" on anaconda command prompt

i m getting errors while installing pyspark on anaconda prompt

@ampcode

Жыл бұрын

Sorry for late response. Could you please let me know which error you are facing?

when executing the function the error is coming like An error occurred while calling None.org.apache.spark.sql.SparkSession. Trace: py4j.Py4JException: Constructor org.apache.spark.sql.SparkSession([class org.apache.spark.SparkContext, class java.util.HashMap]) does not exist. please help to resolve

On this Step I'm getting an error how to resolve this any suggestion from pyspark.sql import SparkSession spark = SparkSession.builder.master("local").appName('Pract').getOrCreate() RuntimeError: Java gateway process exited before sending its port number

@hemanthkumar-ge6rp

Жыл бұрын

same issue

@Saurabhkumar-ve1lb

Жыл бұрын

@@sagarlad5381 already in lowercase, dear

@sagarlad5381

Жыл бұрын

@@Saurabhkumar-ve1lb then the issue might be because of env variables.

@Saurabhkumar-ve1lb

Жыл бұрын

I got it resolved.

@ampcode

Жыл бұрын

Sorry for late response. I hope everyone’s issue is resolved. If not, i’m happy to help😊

Hi Thank In Advance for Your Excellent Video :) I have a question: I am receiving the following error.., What I am missing here and how can I fix it? File C:\anaconda\lib\site-packages\py4j\protocol.py:330 in get_return_value raise Py4JError( Py4JError: An error occurred while calling o41.legacyInferArrayTypeFromFirstElement. Trace: py4j.Py4JException: Method legacyInferArrayTypeFromFirstElement([]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.base/java.lang.Thread.run(Thread.java:1623)

@ampcode

Жыл бұрын

Could you please share me in which step you got this error?

@keshavpatta8797

Жыл бұрын

@@ampcode I to got the same error while running the python code, which creates data-frame, please help. I have one more problem, in my system pyspark and spark-shell command only works when I open CMD as Administrator, will it cause any problem.

@princesadariya7005

11 ай бұрын

I got same error , while I creating a dataframe with createDataframe(), Error: Py4JError: An error occurred while calling o40.legacyInferArrayTypeFromFirstElement. Trace: py4j.Py4JException: Method legacyInferArrayTypeFromFirstElement([]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.base/java.lang.Thread.run(Thread.java:1623)

@princesadariya7005

11 ай бұрын

Have you find any solution?

how to connect a database to apache spark put a clear video for this broo and also for kafka

@loki1076

Жыл бұрын

yes bro i want this video too

@priyadharshinis1798

Жыл бұрын

Much needed video bro pls upload soon

@ampcode

Жыл бұрын

Thanks for your response! Yes we are definitely going to cover how we can connect any RDBMS database using JDBC connector. Please stay tuned and I'll keep you posted on the same. :)

@SenthilKumar-lc8bz

Жыл бұрын

@@ampcode brooo plss upload soon

@ampcode

Жыл бұрын

Yes definitely. Once all the basics are covered, I'll cover this part as well. Thanks!

Sir after installation of pyspark If i run in jupyter its showing as pyspark not installed how to solve this error..plz help me

@ampcode

Жыл бұрын

Hello, could you please confirm if your pyspark command is running from the anaconda cmd prompt. Please let me know so wwe can discuss further.

@harishreeln4706

Жыл бұрын

@@ampcode that error got resolved now its showing as Java object error 🥺🥺🥺

@riyazkhanpatan4602

3 ай бұрын

@@harishreeln4706 I'm also facing the same error. Is that resolved now??

@riyazkhanpatan4602

3 ай бұрын

After adding PYTHONPATH variable as mentioned in the description, I'm getting the following error "PicklingError: Could not serialize object: IndexError: tuple index out of range" Can you please help me on this??

PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number. I'm getting this error while running the code in Spyder IDE. How i can fix this error?

@AnkitSharma9293

9 ай бұрын

Hi there, were you able to resolve this error?

@Karansingh-xw2ss

9 ай бұрын

No I'm not able to solve this error

@pulkitsharma8314

4 ай бұрын

Were you able to resolve it ? I'm also getting same error

@user-jq7de2nf9z

5 күн бұрын

remove '\' from the code and try "spark = SparkSession.builder.master('Local').appName('Test').getOrCreate()" in one line and check . It worked for me

df.show()command is not working

@ampcode

Жыл бұрын

Could you please share me the error you are facing?

But here shows the spark is not defined

Terrible, first you told us to install outdated spark version in last video, now your code does nto work on your installation as it is calling newer version.Thumbs down

I faced the issue with initializing spark object so I had to change it to something like this spark = SparkSession.builder.master("local[4]").appName("Test1").getOrCreate() everything in one line else it was giving me AttributeError: 'str' object has no attribute 'getOrCreate'

Hello, I have tried this, and it didn't work, I am sure of the env variables and everything regarding anaconda env.. I even tried to run Spark on CMD Terminal and changed the Env Variables to point to the local python directory and I was able to get it to work, however I need to run it in Spyder on Anaconda.. can you help me!! Here is my error! Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x6a370f4) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x6a370f4

Hi I have followed your previous video to install and setup all environment variables for python , jdk and spark with webutils, also im able to start spark session from cmd when run as administrator. Now when I installed anaconda and started jupyter notebook for pyspark getting same error as below : Py4JError: An error occurred while calling o47.legacyInferArrayTypeFromFirstElement. Trace: py4j.Py4JException: Method legacyInferArrayTypeFromFirstElement([]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.base/java.lang.Thread.run(Thread.java:833)

@ampcode

Жыл бұрын

Hello, could you please check if you are facing same issues with Spider IDE. if yes, maybe we need to tweak your environment variables. Please let me know.

@lifeitnow9600

Жыл бұрын

Hi I Missed puythonpath env , after adding its now working...thanks for the content!!!!!

@ayushmange

Жыл бұрын

@@lifeitnow9600 hey are you talking about the python path environment variable because i am facing the same issue

@veerabadrappas3158

Жыл бұрын

Hi I am also receiving the same error.., What is the PYTHONPATH you have mentioned ? @Life It Now

@sachinsuman8536

Жыл бұрын

@@ampcode please make a video to solve the above problem

This error is coming while executing : df = spark.createDataFrame(data = data, schema = columns) Plzzzzzz help. I have not been able to learn spark due to this Py4JError: An error occurred while calling o31.legacyInferArrayTypeFromFirstElement. Trace: py4j.Py4JException: Method legacyInferArrayTypeFromFirstElement([]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.base/java.lang.Thread.run(Thread.java:1623)

@sachinsuman8536

Жыл бұрын

Issue Solved. Add PYTHONPATH in your user environment variable mentioned in description above.

@ampcode

Жыл бұрын

Great! Thank you

@riyazkhanpatan4602

3 ай бұрын

@@sachinsuman8536 @ampcode After adding PYTHONPATH variable, I'm getting the following error "PicklingError: Could not serialize object: IndexError: tuple index out of range" Can you please help me on this??

@riyazkhanpatan4602

3 ай бұрын

@@ampcode After adding PYTHONPATH variable, I'm getting the following error "PicklingError: Could not serialize object: IndexError: tuple index out of range" Can you please help me on this??