Apache Spark : How to install Spark 3.4.1 on Windows 11

Ғылым және технология

#apachespark #bigdata #python #java #DET
Please use headset if the volume or audio quality is not good on speakers or try this updated video - • How to install Apache ...
Steps to install Apache spark 3.4.1 on windows 11
Download & install 7zip if not already available in your laptop
1) Download & install Python 3.11
2) Download & install Java JDK 17
3) Download Spark 3.4.1 - pre-built for Apache Hadpoop 3.3 & later
4) Extract the Spark downloaded files using 7 zip
5) Download Hadoop winutil from github
6) Create folder structure - C:\SPARK & C:\HADOOP\bin
7) Copy the downloaded Spark & Hadoop files into above folders respectively
8) Set the environment variables & path for Java, Spark & Hadoop
9) Launch cmd from this path - C:\SPARK\bin - execute command spark-shell
Download links:
Apache Spark 3.4.1 - spark.apache.org/docs/latest/...
7 ZIP - www.7-zip.org/download.html
Python - www.python.org/downloads/
JAVA - www.oracle.com/in/java/techno...
Spark - spark.apache.org/downloads.html
Hadoop winutils - github.com/kontext-tech/winut...

Пікірлер: 72

  • @dominiktokarski8054
    @dominiktokarski805410 ай бұрын

    It's the most clear tutorial on the Internet. You saved me a lot time. Thank you

  • @marcovanderplas2145
    @marcovanderplas21458 ай бұрын

    Thank you so much, after trying several other unsuccessful how to's, this one worked right away :)

  • @vivekdutta5184
    @vivekdutta518410 ай бұрын

    This video is the real outlier! Really helped me a lot!🙂

  • @Explorewithdinesh
    @Explorewithdinesh11 ай бұрын

    Hello Sir. I'm Extremely thankful for this video, I tried multiple ways to install pyspark latest version. but no result. finally, I found your video, thankfully I made it today. please keep doing this type of stuff for future doubts. thanks and regards

  • @dhananjyekumar5339
    @dhananjyekumar5339Ай бұрын

    Thank you so much this video really help me a lot

  • @pieroalonsoveravalverde1747
    @pieroalonsoveravalverde1747Ай бұрын

    Thank you so much! What a great tutorial. It worked really well.

  • @user-ue6gu6uk9p
    @user-ue6gu6uk9p4 ай бұрын

    Thank you! your video helped me a lot.

  • @marysowmyamj8973
    @marysowmyamj89739 ай бұрын

    Thank you so much...this worked well for me

  • @KRPB88
    @KRPB886 ай бұрын

    omg was breaking my head from morning... thanks a lot

  • @balambigaim4828
    @balambigaim48285 ай бұрын

    Thanks a lot for such an informative video. Please post more videos related to Data Engineering.

  • @127_kavita3
    @127_kavita33 ай бұрын

    Very Useful Video. I was able to download Spark and run code successfully. Thanks : )

  • @ganranweiebi1633
    @ganranweiebi16336 ай бұрын

    You just saved a soul. Thanks a million.

  • @darielsolorzanoortega5440
    @darielsolorzanoortega5440 Жыл бұрын

    Excelente video! 😁 gracias

  • @user-ih1hn8ep7u
    @user-ih1hn8ep7u5 ай бұрын

    Thank you so much for detailed video

  • @purnisher98
    @purnisher9811 ай бұрын

    Thnaku sir so so much sir it was really helpful

  • @jyothikajalla6674
    @jyothikajalla66748 ай бұрын

    Good Job! Great Explanation

  • @KuldeepSingh-cm3oe
    @KuldeepSingh-cm3oe6 ай бұрын

    This is really helpful :)

  • @muaz8625
    @muaz86254 ай бұрын

    thanks, it worked perfectly

  • @srinivassathya5098
    @srinivassathya509810 ай бұрын

    Its very useful bro continue the same but increase the volume little more

  • @SufersCarlos
    @SufersCarlos10 ай бұрын

    THANK YOU SO MUTCH !!!!! 😁

  • @nursultankasym8417
    @nursultankasym84177 ай бұрын

    Thank you sir, you are the best

  • @balambigaim4828
    @balambigaim48285 ай бұрын

    Also it would be great if system requirements like minimum memory, processor speed and such details are included for upcoming videos related to installations.

  • @jadeg8621
    @jadeg86216 ай бұрын

    THANK YOU SO MUCH!!!!! YOU SAVE ME!!!!!!!!

  • @georgevayner694
    @georgevayner6947 ай бұрын

    Thank you, my friend

  • @_LAKSHAWADHWANI
    @_LAKSHAWADHWANI3 ай бұрын

    working!!

  • @tarunbhatt5003
    @tarunbhatt50035 ай бұрын

    excellent video

  • @user-zw6ij9tr4i
    @user-zw6ij9tr4i7 ай бұрын

    ❤❤ thank you so much

  • @Turku1-vt8re
    @Turku1-vt8re3 ай бұрын

    Thank you very much for this beautiful video. I downloaded Java 17, but when I type 'java -version' in the cmd, nothing shows up. I couldn't figure out why. What could be the reason?

  • @flosrv3194
    @flosrv31944 ай бұрын

    i have an issue, terminal doesnt find spark-shell even if its into the folder... doesnt work

  • @aidosserikbay183
    @aidosserikbay18310 ай бұрын

    Thank you😉

  • @anshulprakash5334
    @anshulprakash53345 ай бұрын

    you are life saver bro

  • @suhermans7037
    @suhermans70377 ай бұрын

    thanks your share, Sir

  • @meitirii.q
    @meitirii.q3 ай бұрын

    thank youuuuuuu

  • @unluckyme6054
    @unluckyme60547 ай бұрын

    Very useful

  • @manoharghanta479
    @manoharghanta4798 ай бұрын

    How to submit jobs through spark-submit ?

  • @aritra8820
    @aritra88207 ай бұрын

    It is coming as Scala, I need this in python, how to get this interface to be python friendly😊

  • @yusufcan1304
    @yusufcan13046 ай бұрын

    THX

  • @abdullahmohamed5001
    @abdullahmohamed50019 ай бұрын

    thaaaaaaanks

  • @sayantaniguha8519
    @sayantaniguha85192 ай бұрын

    will jdk 22 work? ..or, is jdk-17 mandatory?

  • @ankitkondilkar6648

    @ankitkondilkar6648

    2 ай бұрын

    it wont work

  • @sriram_L
    @sriram_L10 ай бұрын

    Bro for downloading winutils download option is not shown in github

  • @srinivassathya5098

    @srinivassathya5098

    10 ай бұрын

    It's showing on rgt middle bro once check tht it's just 110 kb file

  • @jayanthkumarg8958
    @jayanthkumarg8958 Жыл бұрын

    Bro y in pyspark not every code is excuting, but those codes will run in colab. Can u help me?

  • @purnisher98

    @purnisher98

    11 ай бұрын

    the same is happening with me as well

  • @jayanthkumarg8958

    @jayanthkumarg8958

    11 ай бұрын

    @@purnisher98 bro atleast if you get to know means plz let me know 🥲

  • @DEwithDhairy

    @DEwithDhairy

    7 ай бұрын

    100 % working solution kzread.info/dash/bejne/nINt2byAdda2gtI.htmlsi=lzXq4Ts7ywqG-vZg

  • @DEwithDhairy

    @DEwithDhairy

    7 ай бұрын

    @@jayanthkumarg8958 100 % working solution kzread.info/dash/bejne/nINt2byAdda2gtI.htmlsi=lzXq4Ts7ywqG-vZg

  • @thedataguyfromB

    @thedataguyfromB

    7 ай бұрын

    Python + Java + spark + PySpark + PyCharm Installation Step by step kzread.info/dash/bejne/nINt2byAdda2gtI.htmlsi=aEZ-AM-pGUmaEEVF

  • @ravinithwin189
    @ravinithwin189 Жыл бұрын

    C:\SPARK\bin>spark-shell The system cannot find the path specified. C:\SPARK\bin>Spark -Shell 'Spark' is not recognized as an internal or external command, operable program or batch file. This is the error

  • @prof.mangabhai

    @prof.mangabhai

    10 ай бұрын

    same error , did you solved it ?

  • @bibhutipadhy8736

    @bibhutipadhy8736

    8 ай бұрын

    @@prof.mangabhai bro did you solved it ? if yes ,how ?

  • @prof.mangabhai

    @prof.mangabhai

    8 ай бұрын

    @@bibhutipadhy8736 I think I was having problems with enviroment variables, use this video, my system ran spark when I followed this video kzread.info/dash/bejne/gaGXtbaKhs7YptI.html

  • @DEwithDhairy

    @DEwithDhairy

    7 ай бұрын

    100 % working solution kzread.info/dash/bejne/nINt2byAdda2gtI.htmlsi=lzXq4Ts7ywqG-vZg

  • @taniathakurtannu9223

    @taniathakurtannu9223

    6 ай бұрын

    I getting same error, wasted so much time but no luck

  • @ejazsiddiqui1186
    @ejazsiddiqui11867 ай бұрын

    I'm unable to download winutils file, can anyone please help me out with it?

  • @bhanusri3569

    @bhanusri3569

    7 ай бұрын

    same here.. can someone please help!!!

  • @ejazsiddiqui1186

    @ejazsiddiqui1186

    7 ай бұрын

    @@bhanusri3569 i finally downloaded it I can mail you.

  • @DEwithDhairy

    @DEwithDhairy

    7 ай бұрын

    100 % working solution kzread.info/dash/bejne/nINt2byAdda2gtI.htmlsi=lzXq4Ts7ywqG-vZg

  • @thedataguyfromB

    @thedataguyfromB

    7 ай бұрын

    Python + Java + spark + PySpark + PyCharm Installation Step by step kzread.info/dash/bejne/nINt2byAdda2gtI.htmlsi=aEZ-AM-pGUmaEEVF

  • @thedataguyfromB

    @thedataguyfromB

    7 ай бұрын

    ​@@bhanusri3569 Python + Java + spark + PySpark + PyCharm Installation Step by step kzread.info/dash/bejne/nINt2byAdda2gtI.htmlsi=aEZ-AM-pGUmaEEVF

  • @grzegorzgawinowski3406
    @grzegorzgawinowski34068 ай бұрын

    For those of you for who THIS doesn't work! - If your hostname i.e. local computer has an underscore in name "_" this will stop Spark from launching. Change your localhost name so it doesn't contain underscore and it will work.

  • @arturbarkou6347
    @arturbarkou634711 ай бұрын

    like

  • @user-zw6ij9tr4i
    @user-zw6ij9tr4i7 ай бұрын

    Please share your LinkedIn profile

  • @mainuddinali9561
    @mainuddinali956110 ай бұрын

    u just have skipped many thing ,plz uninstall in ur lapy than try to install step by step

  • @DEwithDhairy

    @DEwithDhairy

    7 ай бұрын

    100 % working solution kzread.info/dash/bejne/nINt2byAdda2gtI.htmlsi=lzXq4Ts7ywqG-vZg

  • @thedataguyfromB

    @thedataguyfromB

    7 ай бұрын

    Python + Java + spark + PySpark + PyCharm Installation Step by step kzread.info/dash/bejne/nINt2byAdda2gtI.htmlsi=aEZ-AM-pGUmaEEVF

  • @satwikreddy8611
    @satwikreddy86114 ай бұрын

    your explanation is the worst, it is literally very slow

  • @TheSunmoon4ever
    @TheSunmoon4ever8 ай бұрын

    Good video with steps. Everything is working fine but for RDDs and createdataframe it is giving error. As given below >>> schema = StructType([ \ ... StructField("firstname",StringType(),True), \ ... StructField("middlename",StringType(),True), \ ... StructField("lastname",StringType(),True), \ ... StructField("id", StringType(), True), \ ... StructField("gender", StringType(), True), \ ... StructField("salary", IntegerType(), True) \ ... ]) >>> df = spark.createDataFrame(data=data,schema=schema) >>> df.printSchema() root |-- firstname: string (nullable = true) |-- middlename: string (nullable = true) |-- lastname: string (nullable = true) |-- id: string (nullable = true) |-- gender: string (nullable = true) |-- salary: integer (nullable = true) >>> df.show(truncate=False) Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases. 23/11/01 20:25:48 ERROR Executor: Exception in task 0.0 in stage 9.0 (TID 9)/ 1] org.apache.spark.SparkException: Python worker failed to connect back.

  • @kollisaikrishna9612

    @kollisaikrishna9612

    4 ай бұрын

    I am also getting same issue

Келесі