I rarely write any comments, but I just wanted to thank you for the content. I've had very basic understanding of webscraping with requests, but recently I've had a project to scrape as many websites as possible and thanks to you I learned about scrapy, pipelining to database, user-agents and much more. Thank you so much!

@JohnWatsonRooney

2 жыл бұрын

Thank you! I’m glad you’ve enjoyed my videos and that they were useful to you, thanks for watching!

Quick note of thanks: I just landed my first programming job which is django development. Before I learned any web dev though, I was always using your scraper tutorials and building my own scrapers; working on my python chops. Those were some of the most formative things in terms of actually getting the confidence that I could learn and write code. I've been quiet on this channel while learning all the web dev stuff but I wanted to say thanks again for all the help you've provided along the way.

@JohnWatsonRooney

2 жыл бұрын

That’s great Matt, congratulations on your new job! I’m really pleased to have helped you in some way on your path to becoming a django dev - thank you for letting me know I appreciate it.

You do a fantastic explaining concepts and give great examples. Thank you

Thanks for explaining everything so clearly!

Very nice tip and intro to Pydantic, I'll give it a try, thank you

John you are amazing, thank you very much for all the knowledge you share, cheers from Chile!

@JohnWatsonRooney

Жыл бұрын

Many thanks!

Best Chanel ever, thank you so much

great work, this is definitely a handy tip

THAT WAS GOOD! Thanks man

You might not know the true impact you are making upon us when making these videos, but it is substantial. I can’t thank you enough for what you are doing.

I've been using pydantic for year. The validator is a killer feature that needs to be built in python next version.

Not in a programming mode but I like watching a master home his craft.

Perfect thank you

I stoped python for a few weeks, and have been learning html,css,js for about 3 weeks. It really help a lot to understand scrapy. Now i'm back to learn more about scrapy skills! I just scrapy all your video titles, wanted to learn one by one.You uploaded 146 videos, a heavy job omg. I couldn't directly scrapy from youtube，maybe there is something wrong with proxy.(live in cn and have to use VPN to get to youtube or google).So I just copy down the html element and use bs4 to parse it.

Thank you so much for this John! 🙏🥳 Great topic. I also watched your SQLModel video: thanks for that too! More videos about storing more complex data structures in an ORM setup are appreciated. My goal is to store scraped data that is enhanced with API data to the Django database. Not sure how to setup the route from JSON + other JSON to the Django ORM yet.

@JohnWatsonRooney

2 жыл бұрын

Thanks very kind!

It's like dataclasses with validation. The only issue I have is that it's a bit rigid. Once it's a pedantic model you cannot change anything inside it. So you need to make the transformations to the incoming data (if any, and yes, there can be) before you instanciate it. But if you're doing that, then you might as well be using dataclasses to begin with and perform the transformations on the go. The problem with that is that you lose the validation, you will have to write it yourself.

Thanks!

I had actually heard of Pydantic several months back but never explored it further, don't know why, just kind of gave it a first glance and moved on. I did not know how useful it could be hahaha I'm definitely going to use this!

Optional doesn’t mean it can be omitted. You need to set a default value for it to truely be optional, but if the default is None then the Optional allows the mixing of list and None

@JohnWatsonRooney

5 ай бұрын

Yes you are right I made a mistake there

amazing!

@JohnWatsonRooney

2 жыл бұрын

Thank you!

awesome

FYI: There is a Pydantic PyCharm plugin that will autocomplete the class variables you type at about 3:33 where you create the item object.

@JohnWatsonRooney

2 жыл бұрын

Oh great thanks I didn’t know about that

You can do this same thing with dataclass mixin module

Not sure if you have tried this, but you can create a top class with a root attribute to hold the list of items. Instead of using list comp. in fact this is what the pydantic Jason to model generators do.

@padraic1983

10 ай бұрын

Like a products model to hold list of product models

Brother, you should make more videos on these libraries other than just web scraping. I love your videos but I couldn't find other library videos.

So is this similar to a json schema validator except that this is operating on a dict? For incoming requests, we're using a json schema validator to ensure the structure and type match against a json schema. I'm wondering what this will bring to the table should I replace schema validation with pydantic.

Nice 🤠 but I Have a question how modify the Json (e.json) when the exception is raise

I had literally just said aloud "yeah, I don't want to type all that out" and then you said "I can hear you saying ''I don't want to type that out all the time". :))

@JohnWatsonRooney

Жыл бұрын

Haha yup!

Hey, what tech stack do you use for your blog?

@JohnWatsonRooney

2 жыл бұрын

Hugo static site generator, GitHub and netlify

how about taking it a step further and doing using dataclasses? Instead of using basemodel? Would that work too? from pydantic import dataclasses

Hey John, are you going to do more tutorials on Go? You are a Gofer now after all 😁!

@JohnWatsonRooney

2 жыл бұрын

Haha yes I will, don’t worry

You can achieve all of these with dataclass also

John, pls explain what does it mean in list comprehension this Product(**item)?

@JohnWatsonRooney

2 жыл бұрын

We are loading the data into the Pydantic model called Product

@vitalij09

2 жыл бұрын

@@JohnWatsonRooney thx

@dolamuoludare4383

Жыл бұрын

@@JohnWatsonRooney I guess you are trying to unpack the dictionary/json data structure item?

for example u have 1 excel sheet and it consist of 10000 data in it. Later when we import that excel file in pycharm or jupiter notebook. if i run that file i will get an Index range also know as Row labels. my python code should be able to read that ten thousand row labels and should be able to separate / split into 10 different excel sheet files which will have 1000 data in each of the 10 saperated sheet. other example is, if there is 9999 data in 1 sheet then my python code should divide 9000 data in 9 sheet and other 999 in other sheet without any mistakes. i am asking this because in my data there is not any unique values for my code to split the files using .unique plz help i have search the whole YT , stackoverflow, and github tooo from 3 days

Love your video and learn today. I am not clear of at 9:37 product[**item] , will you please explain what "**" mean here in the list comprehension. Look forward to your reply. Thanks

@VolundMush

7 ай бұрын

In Python, ** is used to unpack dictionaries/key-value pairs. This is useful in a few areas, I mostly see it used in function/method parameters. def function(*args, **kwargs): would let you call function(blah=True) and kwargs would be a dictionary that looks like: {"blah": True}

@alecd8534

7 ай бұрын

@@VolundMush thank you so much

I wanna thank you for teaching the world. Please I'm python beginner I my next step is to build web scraper.can you please help me how do I go about it.

@JohnWatsonRooney

2 жыл бұрын

Thanks! I have loads of web scraping with Python content on my channel if you have a look I did a basics scraping guide not that long ago

I find this video shallow and pydrntatic -Peter Griffin

why not simply use a dataclass, they have types as well.

Ha ha python is moving towards java...

Don't show off your typing skills, but use copy and paste. I find it particularly annoying to cancel and correct your typing in my mind, while you do it!

Great content, but watching someone type is annoying AF! Why not type it up beforehand, then hide and reveal lines as you go?!

@JohnWatsonRooney

6 ай бұрын

Different styles, but fair enough - my typing skills aren’t the best

@validator("sku") def sku_length(cls, value): assert value == 7 return value must be better

@JohnWatsonRooney

Жыл бұрын

I’ve always worked on the assumption that assert is for use in tests not code logic

@oktayozkan2256

Жыл бұрын

@@JohnWatsonRooney fair enough