👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis

I would have loved to see your approach to logging from multiple processes in the same files. For example logging from a FastAPI application that uses uvicorn or gunicorn with multiple workers. Perhaps in a future video. Looking forward to that :) I really love watching your videos. They're full of useful information.

@abhishekpawar921

Жыл бұрын

+50

@thomasjohanns7661

Жыл бұрын

Or from multiple files/modules. I've written a function that sets up logging and call that one in every file, but have no idea if that's the way to go.

@_DRMR_

Жыл бұрын

@@thomasjohanns7661 You typically only run the logger in your main function. Then in the config you can set up how it should behave for different python modules (including any imported modules. say if you want to silence, or actually increase, logging for anything in particular).

@glennlehman9608

Жыл бұрын

@@_DRMR_ While this is a common practice, it does not always get implemented this way. For example, if I use a custom library that might have its own logging module. Also, multiple processes could imply two or more handlers on the same logger.

@TomTrval

Жыл бұрын

We use initialized loggers. Do not forget to check official python build in: QueueHandlera and QueueListener for separation of handler (OUT end of queue) from logger(INPUT side pf queue)

As it is named deep-dive I was a bit hoping the focus of the logging levels and what are the daily praxis where to log, and how to handle the multiple levels in the code. This is something after years of developing software I still struggle with. Sometimes it feels like every second line is a log, and sometimes there are 3 log lines in the full project (hyperbolic). It never found a good regular way / a principle to follow ... :) Whatever! :) Keep going, I learned a lot from you the last year I have now in my daily praxis. It helped me a lot :)

@Lodinn

Жыл бұрын

I've found talks by Kevlin Henney incredibly insightful and interesting. Here is a part from a talk (with a clickbaity title) where he addresses logging: kzread.info/dash/bejne/eK13u9iaaafOo6w.html. No concrete advice, but hope it's helpful nonetheless. My personal idea of logging is pretty minimalistic: YAGNI. I make extensive use of logging in the debugging phase, as regular debuggers are often of little help with kinds of software I'm dealing with. But after I have figured out some part of the code and it is covered with tests, logging goes out of the window, unless I have very specific scenarios in mind where I would need it for a post-mortem. For example, if I need to log user actions, I have a set of requirements for that. Otherwise, I do not need to log user actions.

Great video as always! Interesting to see use of rotating logs as well as multiple package log implementation. I know I ran into issues with those when I first started writing in python.

You're videos are absolutely brilliant but what I really love is the code being available. It saves me a lot of time when I come back to one of your videos I've already watched a couple times.

I’m a terrible coder, but only do it for personal projects, and I never thought about using logging to replace the print statements I use when developing, this would save me having to find all the random print statements all over my code 😊

Fantastic as always, would love a deep dive on the structlog library for your next video! really takes logging to the next level

@ArjanCodes

Жыл бұрын

Great suggestion!

Logging throughout a module is way more complicated than you’ve shown though. Can you show how you set that up?

@walterppk1989

Жыл бұрын

Was about to type this

@TheCarmacon

Жыл бұрын

Yes please!

@akashdesarda502

Жыл бұрын

100 % agreed

@dirtdart81

Жыл бұрын

i just put "logger = logging.getLogger()" at the top of each file and call logger as usual. Works great

@allo5668

Жыл бұрын

@@dirtdart81 but then you’re always grabbing the root logger, which is bad practice as far as I can tell

Yessss I’ve been waiting for this! Thanks Arjan!

Thanks Arjan! Could you do a follow up on structured logging?

I feel as though you just glossed over some of the basics and added the SysLog as a bonus. What about getLogger(__name__) combined with propagation? What about different handlers at different levels? What about custom formatters? What about using decorators to do your logging for you? If nothing else, there are some ideas for a future show :)

Great content as always Arjan! I think it would be also a good idea to show ways of not overly littering your code with logger calls (ie. using decorators, as in one of your recent videos, or perhaps using other methods) and perhaps demonstrating a real-world example of crucial logging within an app. All the best!

Nice. I would have loved if you had shown what's best practice to actually log and at which places in the code loog statements are supposed to be.

@hawks3109

11 ай бұрын

That's very difficult to explain and is up to the project. It's almost artistic. The best way I can describe it is log anywhere you think will help you understand what the software did without tracing the code. You want to be able to diagnose what a problem is almost from the logs alone. That expands to log any information you need to diagnose. Like user names or values that were switched on or errors encountered

waiting for this!!!! Thanks Arjen!!

Great job! Thank you so much

Would love to see more advanced logging concepts especially unified logging in a big library

I also find logging super important! Thanks a lot for this video with lots of good advice :)

Thanks, Arjan! As always, amazing video!

@ArjanCodes

Жыл бұрын

Glad you liked it!

I love your videos about theory and architecture....once you have been coding for a while and you are no longer a begginer this kind knowledge feels really escential.

Arjan, thanks for uploading this :) the date/time formatting of logs was very helpful. I've been using INFO:ROOT this whole time like a n00b.

As always, excellent video. Making a simple topic so robust, I want to build some apps just to log now.

Amazing content Arjan, keep it up. Cool intro to Paper Trail too :) Been playing with log2d that makes logging a bit easier.

This was a lot more basic than I was expecting. Python's logging has a lot of pitfalls that I would have liked addressed.

Again an excellent video ! I'm actually pointing all my new devs to some of your videos so that I'm sure they pick up good practices (even old ones like me). I didn't know papertrail, and I'm now starting using it. it's perfect, though I'm a bit worried about anybody being able to send logs to my account (no real auth)

Yeahhhh thank you so much!! Really appreciate a logging tutorial!

@ArjanCodes

Жыл бұрын

You're very welcome!

*Thanks Arjan!* That is super useful 👍

@ArjanCodes

Жыл бұрын

Glad it was helpful!

I am glad to have found this video. Thank you!

@ArjanCodes

7 ай бұрын

Glad it was helpful, William!

I've used grafana + loki for very long time. Its very efficient with little to no resources. It can be setup in multiple ways and there is also a package for python called python-logging-loki. Loki has a really nice and powerful logql language with which you can make awesome grafana dashboards.

Super useful, thanks!

@ArjanCodes

Жыл бұрын

Glad it was helpful!

as always, great content Arjan.

@ArjanCodes

Жыл бұрын

Thank you so much!

long-awaited topic. thanks!

Thank you for the video. I didn't know SysLogHandler and Papertrail before. In our company, we are using Graylog instead of Papertrail. Graylog isn't only for Python apps but also for other apps creaded via Java etc. Graylog has a REST API Server. Thus, we are accessing logs in Graylog via requests library in Python. Then, we perform Fraud Detection based on logs data. There are also some libraries like loguru and structlog for logging in Python.

Great Job bro !! Keep it up.. Thanks for this ❤🙏🏻

@ArjanCodes

7 ай бұрын

Thank you for the kind words! Will do.

I would agree with other commenters, it's nice to get a basic intro to logging, but I'd really like to know WHAT I should be logging. In theory I could put a debug log for every line of code I write stating would should happen, but I imagine this is overkill. Or is it? I don't know where to draw the line, it would be good to get some examples of what kinds of things you log, and what level is appropriate for them. But good video, and I'm looking forward to a part 2 😉

Thanks 🙏🙏🙏

Always good videos

best for someone who just started to learn logging.

another important use is audit logs... we have some critical operations that the customers have to be able to see when they ran, how many times, with what parameters etc

You are doing great videos, but IMHO this was a bit too basic, would love to see an even more deep dive into this module. Like setting it up for a bigger project, best practices, etc. Anyway, thanks a lot and have a nice weekend! 🙂

@OhsoLosoo

Жыл бұрын

I think its good he started off basic so those of us who dont get a chance to use this stuff can have a basic setup. I hope he makes an advanced one too!!!

@aiml84

Жыл бұрын

I am also looking for setting it up for bigger practices and best practices. i.e. I also would like to know how you can pipeline the logs to may be ELK or anything where it can be consumable

@kayakMike1000

Жыл бұрын

You gotta start somewhere. I loved it!

@ianrickey208

Жыл бұрын

+ 100 on this idea. Most logging ends up being a overflowed garbage can of rotten log salad and parsing it to find confirmation of correct application, business logic performance, and the mixed bag of errors is just difficult to retrofit after integrated into production. Once it rots in production awhile, logs are definitely an 8th "olfactory offense".

@user-us7rg4cd6p

5 ай бұрын

@@OhsoLosoo Title should NOT be misleading. This is amateur/beginner logging, pretty far from Pro.

Some logging services like Splunk provide a custom query language, and it's worth to learn it because it will save a lot of time. Also another tip is to avoid root logger and prefer to create a logger for each module, it will reduce the searching time as you can do more specific searches.

Thanks Arjan, good start on logging. I recently struggled a lot to configure getLogger to get logging object from other module. Python documentation is not that clear on this. I even tried to use thirdparty pkg like logguru. Try doing a video on configuring multi file/module logging and best practices

Great video logs simple and insight full. But this content is available like everywhere, can you make a video on logging with different modules in the project or when using a design patter in projects and we have tons of files and folders in our project.

I really appreciate your efforts in making this video and it was enjoyable to watch. Informationally however it doesn't quite live up to the expectation set by the title...

I enjoyed the video. I am currently working on doing a presentation on logging. During that process, I spent a lot of time understanding handlers and formaters. I become most unclear when looking at the best way to implement logging in a package compared to how to implement it in an application. Do you have the link to the second video you mentioned on logging?

I like you content very much. I'd like to know when to use logging & when to use print statement. Currently, I tend to checking "logging" via print statement.

DUDE.. I was searching for Logging tips TODAY.... lol

Hi Arjan, thanks for this video! I was hoping to learn how I should set up logging for a package, so that people that might install my package via pip get access to the logs when executing the code if they wish so, but maybe not automatically. Do you think you could make a second video about more advanced nuances about logging? Also I saw another package (requests) set a NullHandler in __init.py__ and I was wondering what the reason behind that is.

you are best my friend.😍😍😍😍😍😍😍

We have struggled with logging quite a bit in our Company and I guess the best solution if to use a central logging config object (dict or even better yaml file). There u set the Formatter, Handler and if you need to set custom attributes, you can do that pretty nice by using a filter.

@NostraDavid2

Жыл бұрын

Yesss. Log as JSON, ingest those logs into some central service (you could use Filebeat for ingesting the log files and use Elastic as search engine with Kibana for dashboarding and searching through your centralized logs). You can also use `jq` (a terminal application) to sort, filter, format your files, etc. It's sooooo nice.

@NostraDavid2

Жыл бұрын

I use the structlog lib, which can be a pain to configure, but it's incredibly powerful and flexible.

@codingcrashkurse6429

Жыл бұрын

@@NostraDavid2 Yep, we also use JSON format for the logs and a central instance where u can see all logs. This is luckily done automatically in our Kubernetes Cluster

Thanks for the video. Looks like papertrail is now solarwinds, any idea how to do the port stuff with the new set up? Would be great if you have any suggestions

Nowadays one of the main reasons for a logging server is separation of concerns in the light of information security. If you get attacked, you may want to have a log server separated from your operational data proccessing - with no possibility to remotely delete your logs. Forensics will be very thankful.

Some tools (which can be a bit overkill) like splunk and elastic stack can really allow you to search and filter log files with ease. But this however is coming from a digital forensics background.

Hi there, from Portugal, Any recommendation on using python for google cloud function using google cloud logging? Obrigado(Thanks)

I expected you would mention log data has an interface. Changing your log statements can have serious impact for tooling that analyzes the log files for statistics, problems, and improvement opportunities.

I agree that this was too basic. Mentioning Paertrail was very helpful

A great video and explanation.. Do you have courses in Python from intermediate level?

@ArjanCodes

2 ай бұрын

Yes I do! You can see all of my courses here: arjancodes.com/courses/

nice

Logging to splunk is useful too.

With normal file logging you can search with grep, less and all the other standard tools, what could be easier than that?

Good introductory video about logging in general. However not a deep dive on Python's logging package, as announced at the outset. In fact there is surprisingly little about Python in this video.

Hi, I am using this functionallity, but module "requests" messes up my log, as it also throws its requests, and for example I see updates of virus signatures and configuration downloads from firefox, is there any way to prevent it?, I could upscale my messages as warning, as it seems it only happens in the INFO configuration, but there has to be another way. Thanks for your great videos!

Valeu!

@ArjanCodes

4 ай бұрын

Thank you so much for the support!

I used nbdev in VSCode to write python modules. Where does the logging module write it's output when working with notebook cells?

@thedrumify

Жыл бұрын

# import reload from importlib import reload reload(logging) logging.basicConfig(format='%(asctime)s %(levelname)s: %(message)s', level=logging.DEBUG, datefmt='%I:%M:%S')

I like the sweater, what brand is that?

IMHO the logging module seems really over-complicated and easy to get lost in (handlers FTW). But now that it’s in the std lib, nothing more Pythonic gets much traction. See at all the comments asking for complex configuration examples, across multiple files. Video is a good intro to its existence but there’s only so much that can be conveyed this way. And in blog posts as well, most of them also barely scratch the surface of Python logging.

Why did papertrail mark the log message as coming from "logger" when the logger you configured was named 'arjancodes'?

I wonder how often do large companies use an in-house logging solution. I’d imagine that would prevent some of those data leaks and would be perfectly feasible to implement given their size.

Im working in how to implemet logs with fastapi and just I realised logs are printed in stderr by default wich will be a problem if a cicd pipeline relies on stderr in order to continue the right flow. I hope you will explain it here :(

Here's a great example... You can have python logging framework output to a syslog server... It's AMAZING. Imagine a few hundred users using your stuff and you get all the usage info, who is crashing and so on and so forth.

👏🏾👏🏾👏🏾

how do I log to system.log on macos ventura? none of what i've found on SE or what has been provided by chatGPT works (no errors though)

those unequal length cords on your hoodie is driving me crazy

I'm liking the glass of water 🤓

Logging service, you just blowed up my mind :) I had not idea it existed. As usual thanks for the great content.

@ArjanCodes

Жыл бұрын

Happy to help, Daniel!

My question is if something goes wrong in production.. how can we back trace the problem from logs?

Really important! Why is getLogger etc from the logging package camel case? It hurts my pythonic eyes

Thanks! I am trying to force people to use logging framework on a company internal project, it packed full of prints and I am so sick of getting screenshots of a command prompt.

It'd be cool if you showed logging config from different files, like ini and yaml. thanks!

👍👍

what is the name of the compiler?

Why at the first place you need a separate logging service.??

@ArjanCodes

5 ай бұрын

All your logs are in one place, so you can more quickly find problems, better log visualization features such as colored logging and various types of charts, better log search, better export and backup capabilities, better automation features. Just to name a few things off the top of my head.

what is the purpose of the "if __name__ == "__main__":" block

/dev/null as a Service!

Me, an intellectual: print(x) scattered throughout my spaghetti code

Real talk where is that sweater from?

This dive was into this module was really shallow

Anybody here uses loguru?

Logging into a File with a Rotating FileHandler from multiple processes is not so nice tho xD

nothing compares to print("here")

Logging is hella important. If you think logging and not well organized and built logging systems is not important, you're still very junior.

Log to a database ..sqlite3 baby

remove all of this pain with just `pip install loguru` and `from loguru import logger`... yw

I get a bit depressed when I watch videos where I already knew everything that was explained. It's nice to watch it accurately and well explained but like following a football team you're hoping for a goal, that piece of knowledge you didn't have, but that never comes. After a while you reach the end with this programming thing and you learn that it was the learning itself you loved, not the programming.

This type of logging is insufficient. Where's the line number? Where is the thread ID if your app is multi-threaded? Where is the task ID if it's asynchronous? How will you know in which thread/task the error occurred.

How do you configure your emacs this good? Can you make a video about this as well?

It is good，but i choose loguru😂

Big mistakes with logging: Do not over-log information, just what needs to be logged. You kind of said this with the security part, but still.

Auto translation of the video title to german totally butchered it.

@ArjanCodes

6 ай бұрын

Yes, we switched that feature off for more recent videos. It doesn’t work that well.

Title should NOT be misleading. This is amateur/beginner logging, pretty far from Pro.

this was hardly a deep dive. this was VERY surface level