Hope you enjoyed! ❤️ Don’t forget you can try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/TheCherno . You’ll also get 20% off an annual premium subscription.

@code_with_MK

21 күн бұрын

Sir , continue the C++ playlist if not then tell talented teacher for teaching in more advance and has been better as you.

Serialisation is a topic I've wanted to see a video on for so long, I got so so excited when I saw this video was released!

Text format design is very human

@mr.cobalt6695

20 күн бұрын

'the design is very human'

@tawandagamedevs

20 күн бұрын

😪

@MehmetSarkd

18 күн бұрын

at the cost of speed

One thing to remember is that when using "plain" text that the encoding and file formats still matter quite a bit too. For instance, whether you use ASCII or UTF-8 can make a huge difference, and while YAML is an okay choice, TOML would be better because whitespace really shouldn't be significant in a user edited format.

Ah, that brings back the nostalgia of opening an .ini file of an old game for the first time and changing things just to see that it actually changes what the game does. It broke things more often than not, but that was my first interaction with the concept of variables.

The best solution is USE BOTH. On my current project(s), I use only binary because they are small projects. If I had more time, the level files for example would be saved as xml or json. That way merging is easier in git, and if I break loading/saving code and save a broken file, I can edit the broken data manually. Then I'd write an exporter to export those text/json/xml into binary when I export the project data. Only the problem now becomes that you need an additional QA pass to check that binary data also works. Although, player options are good to export as xml, so users can change things. Which is what I used on my previous project. And if you're going to use binary, as ThePrimeagen said recently on his stream: ALWAYS start with a version at the start of that binary format! I'm using this in the current project and I can modify the saving/loading code for level data and still be able to load an older level to save it in the new format.

I was just looking for this. U really read my mind. Thank you so much

I've only ever done binary serialization in C/C++, in Go it is super easy to serialize to text (JSON) so I use that in most of my Go projects. Interesting video!

Can you do json vs yaml? Which is better for storing data, readability, commenting, which is better to use as config file? Are there better formats ?

@DemoBytom

21 күн бұрын

in most cases json and yaml are pretty much exchangable. Technically yaml is a superset of json, as in it has more data types supported out of the box, and supports things like references within the document, but overall serializing/deserializing it will, in most cases, be more or less the same. In the end both are key-value pair, human-readable documents. Readability is personal preference. I saw people preferring yaml, but personally I vastly prefer json format, but tbh.. But that's very personal opinion.

@Brahvim

21 күн бұрын

@@DemoBytom For some reason I feel that YAML must be easier to parse, given that it doesn't exactly use braces like JSON.

@DemoBytom

21 күн бұрын

@Brahvim nah, for a serialer it would be easier to parse json than yaml. Json doesn't care much for indentation, nor white characters, while if you screw indentation in yaml the whole document is invalid. Granted - both are pretty much non issue anyway, because both json and yaml can be validated by tools to be correct, and I in prolly 99% cases nobody writes their own serializers/deserializers for them, since better ones already exist in any language/framework already.

@anon_y_mousse

21 күн бұрын

@@DemoBytom Just use TOML, whitespace isn't significant since it's basically just an extension of old INI files, but it adds some extra functionality.

@asandax6

20 күн бұрын

Use TOML and stay away from YAML (Curse you flutter) and use JSON if you don't want TOML.

Nice comparison, I would love to see the code as well in a future video!

Great Video! Can't wait to see the follow up. Keep up the good work!

This and memory management is one of my favorite topics in programming. Thanks for the video!

Another reason for using text format is if your software runs on multiple hardware configurations/operating systems. In web development it's common practice to use json serialization for almost everything.

Im working on a game engine and am currently using binary serializing, similar to how you would if you were pushing/reading a stack (ie packet deserializing) and integrating a reverse dump so the runtime can be the editor

13:27 Absolutely! That's the kind of stuff I work with atm with modding, and optimisation is always something I'm seeking to learn

My best programming and my favorite teacher in my whole Life.Thanks to you Sir very Thanks....

In the game I'm currently working on we store quests as json in the editor, but use MemoryPack for the runtime

Nice vid! Always has thoughts on this topic - Unity serializes in YAML text file, UE serializes in binary. Personally I think the most ideal engine would store assets in text format in editor and during cook process would generate a binary version for that build for smaller size and some "encryption" of data, merging blueprints sucks merging scenes in Unity is somewhat doable from my experience

while it still comes down to whether the format is intended to be read by a human or not, another reason/side effect for using text formats in the shipped build is modding on i can still remember is in GTA San Andreas the data for vehicle specifications was in a text file (csv iirc) and allowed for some fun things like insanely fast tank while determined users will still find a way to parse and + attempt to edit anything the text serialization does make it easier which depending on whether the game devs want to encourage/discourage may be another factor one way or the other

Personally, for hexeditors, I prefer imhex, because it provides a sort of scripting interface to automatically annotate data in the files.

For me one more good reason for text is future changes in the data format. If you add/remove some variables in your format text parsers will easily handle old versions by using defaults/ignoring some keys. But binary ones will need added logic for each change and there needs to be version of the format saved in the data. Of course you should always have format version in all data even the text ones.

Can’t wait for the next video. Let’s get it!

This is a good video..what about a video about cpu vs gpu in game programming. Can gpu execute algorithm? Are they only used for vector multiplication?

@CD4017BE

20 күн бұрын

Modern GPUs can mostly run arbitrary code but there are some restrictions: - no recursion (the maximum size of the call stack must be known at compile time and is very limited) - no indirect function calls (for the same reason as no recursion) - no dynamic memory allocation (so you can't use things like lists or hash maps) - branches are potentially slow (because multiple vector cores get fed from the same instruction stream and can't execute different instructions at the same time) - sequential code execution is much slower on GPU than CPU so in order to achieve a performance advantage, you must design your algorithms to run heavily parallelized. There might be some more things I haven't mentioned here but these are the main limitations for GPU algorithms.

Haven't needed to do anything with binary for files in a while, largely setup any config type files as text exactly for the reasons described in the video. However if I had to make a binary file format now I'd probably consider using something like Protocol Buffers, mainly for maintainability.

Does Hezel support something like Unity's Addresables system? I'm meaning a system that allows content to be packaged separately from code into packages.

Use binary if you wana store some data (which is not needed to be used by hummans) it can be effecint at reading time, storage etc While use some Text if it's humman designed,it will be slower but readable You can use JSON, or custom fornat or any other Theres tons of them

What notes/whiteboard software did you on this video?

Binary files contain data of a fixed length, and text files are also binary, with a lot of line breaks and a lot of human readable numerical values, which must be converted back from ASCII to numerical values ​​that can simply be placed in memory.

could you serialize to text during dev, then automatically serialize to binary during production, so during dev you can easily edit the save file, but in production it is just stored as a .dat

Informative video. I would love to see the code as well :)

Not packing the assets in the development phase is definitely understandable, however I believe keeping all data in binary on the disk is fine. Instead of keeping the files in text and parsing them on runtime, you can also parse the binary in the Game Engine editor and show the developer a text version or any kind of human-readable UI which gets automatically converted and saved back to binaries on disk. The big downside here is that extra work needs to be put in to create a nice editor, but it may be a nice middle ground for some scenarios.

ASCII is binary data as well; it is just interpreted/read and/or written differently.

quite useful

I'd be interested in seeing how a GLTF/FBX/OBJ can be converted to an "engine-friendly" format for faster loading.

@custard131

12 күн бұрын

i will use OBJ as example because its the one im most familiar with thugh i expect the same logic applies for most other file formats when you load an OBJ file into an engine the engine is going to have to loops through the data in that file and build up an array of vertices in memory with whatever data you have for each vertex (eg position/normal/UV coords) one option that would be relatively simple and i think would result in the fastest loading would be to simply dump the resulting block of memory out to a binary file, then rather than needing to perform any processing at all you could just load the file into memory and point a variable at it for a setup where the originals are in text based format and only the shipped version is binary that feels like it would be pretty close if not an exact explanation to what is done

Finally, we are getting there

If you need serialization there is nothing better then chunks with headers that you just memcpy + relative pointers

I feel like the merging and version control point is the most important part of this. Git, SVN, Plastic... Whatever you use, it doesn't like binary files and will likely not be able to merge changes of two contributors into the same file in your repository. If you're working in a team, trust me, this becomes a huge nightmare.

It would be really cool to see the implementation differences between text and binary serialization. Do you use a lot of bitwise operations with binary serialization? Is bitwise logic actually faster than standard logic in C++? I have wondered about these things for a while. Lol. Thank you for your videos!!!!

@malekith6522

20 күн бұрын

It depends on the problem you're trying to solve. When you're dealing with communication (protocols) where bitwise operations are very much in use and you need to splice your number into bits, then bitwise operations will be a faster and better choice by far. It's just a tool that could be useful, and there is no separation between bitwise logic and standard logic of CPP

Amazing Contant , always Unique.

I would love to see a vidio of the serialization code

do the shipped files get enrcypted or otherwise protected against "malicious" changes/extracting game assets besides being in binary

@TurtleKwitty

20 күн бұрын

fun fact your user needs to have decrypted files to use them so it literally doesnt matter in any way

@custard131

12 күн бұрын

a waste of time both in terms of development time trying to implement it and loading time having to decrypt at least for the stated goal if you encrypt the assets, then you would also have to provide the user with the key to decrypt them so they could play it not with games but i have seen people try to do this exact thing with software and its a complete waste of time in almost every scenario except for malware avoiding detection

@qqshutup7175

12 күн бұрын

@@custard131 Most games that don't have file encryption are hacked, attributes are modified and can generate a lot of strange behavior on the client or server itself, assuming that the client/server will trust the client's information "a little". e.g.: A weapon has an ID and the client sends that it is using that ID, but the weapon's ID has been modified on disk and now the server "thinks" that the client is using another weapon, that's it, you've exploited a problem... Many games encrypt and decrypt files in parallel, others prefer to hash the file and send it to the server to confirm that the file hasn't been modified. If you don't do either of these things, your game will be destroyed by cheaters. If it's an offline game, no problem, but an online game can't do without this.

@custard131

11 күн бұрын

@@qqshutup7175 that is a problem that needs to be solved for online multiplayer games, but encrypting of assets doesnt solve that in such a setup the server needs to keep its own copy of the attributes and use those rather than relying on ones submitted from the client there is essentially nothing you can do to prevent someone modifying how thing appear in thier client, but in realising/accepting that there are ways to design it in such a way that it doesnt impact other users signed data packets can be useful in some cases but i see that as a completely separate topic to encyption the simple fact here is that in order for a user to be able to play a game where the assets are encrypted, they are going to need to be able to decrpyt them

Yes please make a video on binary SerDes!

I would like you to discus xml reading writing which are useful for small configuration data store.

@christianm4906

20 күн бұрын

Almost everyone in is moving away from xml and adopting JSON instead.

@malekith6522

20 күн бұрын

@@christianm4906 Agree, usually on the team, we will never approve the use of XML if we are not forced to.

I'd love to see how you do good Serialization in C++ Have tried many times, but never got a satisfying and safe result

I just realized that serialization is really not that different from code compilation. Just like when you compile your human readable code into machine code, you compile your human readable text into a binary format. (Serialize your values into a binary format.) That’s all it is. Is it for humans or machines? Your choice will make it easier for one of them. Make it for humans and machines need to translate. Make it for machines, and humans have to translate.

@urisinger3412

21 күн бұрын

it really is diffrent, serialization is supposed to be lossless, while compilcation is lossy. compoilcation is more like deserialization, which is more lossy.

@njnexgen

20 күн бұрын

​@@urisinger3412 ahhh so your reply was compiled then

@tedchirvasiu

20 күн бұрын

@@urisinger3412 Combobulation

@tedchirvasiu

20 күн бұрын

@@urisinger3412 what is being lost durring deserialization?

@urisinger3412

19 күн бұрын

@@tedchirvasiu depndes how you serlize it, for exsample json deserlization losses some information like indentation

1:35 no there is no link in the discription

I don't understand, why(at least in 2024) all the code editor and even notepad decompile or decode idk then compile or encode when you're done editing, so everything stays in binary, it's just test surely it the cost is negligible right?

A thing to note is text serialization is really not that bad on strings, it's only in numbers where the perf can get a bit worse.

when I serialize gltf model to binary it loads extremely faster but it turns from 70 mb to 300 mb on desk !!!!

Sir pls start a series on vulkan with c++

Why did you go with YAML

The idea of content files not being human readable on a released product hits a snag if you factor in Modding, if you want games to be mod friendly a large part if not all of your config style content files (aka not 3d models or texures and so on), should be human readable, but, yes, that increases load time, so why not have the best of both worlds? provide a "development" build, might also include more verbose logging, command console, asserts and content / asset validators, which works with text files as well as the release build that pre-processes those files once into binary and from then on will load faster? remember, any modding support and / or tool provided will also likely benefit designers and content creators on the dev team, so everyone wins

for text data that you don't want to be text data in the published game, why not just represent it in some entire c++ constructor (or other language) in some file? that way its human readable, but the compiler does the job at serialising it for you

wouldn't be great if we could go from text base programming to something even easier for humans like something visual or something we can't figure out yet... I always thought that when I was in college studying this stuff. 😆

@asandax6

20 күн бұрын

You mean nodes? Like the ones they use in shader scripting or visual scripting.

You didn't link the hex editor!

@TheCherno

21 күн бұрын

Thanks, added!

I'd put my cutoff at around 100MB. If you can save the file as a JSON string or other text format in less than 100MB, go for it.

👍

Modding text files is easier. Binaries will need exporters or converters

I do text most of the time. Binary is pain in the ass to extend

if"other humans come in and change it"... very alien like

I think this novel introduction of background music is kind of distracting, which in my opinion is not good for the content of this particular channel. Please take that as a constructive suggestion. Said that, just to share my experience as a retired software developer: storing and loading application memory models in binary is virtually instantaneous but slightly more prone to corruption due to subtle bugs. Also versioning is more difficult to handle. On the other hand, text based memory models are arguably slower but a lot more robust against bugs (particularly versioning bugs). But please do not use standard json libraries or other library based stuff because these tend to be slow as hell. Just do your own parsing and you will be fine and a lot faster than ANY existing serialising library. In at least one occasion I chose to use both alternatives simultaneously in a single file bundle. Saves created both formats, but opens only used the binary form. However, the text version was always there as a fallback in case something went wrong as reported by the appropriate data integrity checks. But I mostly developed document based apps, not games, so these files were updated by the users by using the app, adding further risk to their integrity, and the need for robust versioning. Otherwise, I understand that loading heavy textures and 3D models for a game is just too much demanding for text.

I use binary serialization in my game engine, similar to how unreal engine does. I prefer binary serialization because generally, it is faster to deserialize binary data.

@paradox8425

21 күн бұрын

That's old school and one of the reasons I kinda don't like UE anymore (Even though I regularly worked with it for past 5 years). 1-) You are forced to use editor at all times which is usually not open since you write cpp 2-) Since it's a development environment frequent changes and crashes occurs. This sometimes ends up in binary corruption without any way to recover. If it was a text file, at least I could see what went wrong to prevent it in future. I might even be able to recover 3-) Using text files doesn't meas you can't use binary files. A good approach is storing only metada as text (depends on asset type)

@neil_m899

20 күн бұрын

@@paradox8425 well I already made my custom binary asset serialization format LOL. I'd rather focus on the engine itself. Plus, the asset loading (deserialization) in my engine is super fast. And writing 2 different asset formats (text + binary) is too much for me. Currently, I have a class called Object, and a class called Package (which is subclass of Object). An object can have children objects in a hierarchy format. And Package is the object that is serialized to disk with all the children objects serialized under it. So it's very much like Unreal engine lol.

@paradox8425

20 күн бұрын

@@neil_m899 You can do whatever you like. What I'm saying is, reliable and easy to use tooling is much important than most of the people realize. Sure, it might not increase performance of shipped game, but it can make you ship it 2x-5x faster or maybe even more depending on the exact situation. You say "binary assets is fast", but did you benchmark it with the text assets? Exactly. Just like cherno said most difference it can make is 2-3 seconds to load an entire scene which is usually differs much less. And by doing that you gain ability to review chanhes directly from source control and edit without editor open

@vaijns

20 күн бұрын

An important thing to evaluate about performance tho, is how often will the reads happen? If it's just once at startup where you have to wait say 1 second instead of 0.1 seconds, who cares (of course if it's many files, that adds up)? But if it's inside some kind of loop that's executed regularly this difference does matter a lot (best is no file io in loops at all ofc).

TOML. Use TOML

Sometimes libraries force you to choose between these two formats

During development I'd use text files .. but I'd never ship them .. just to prevent users from screwing around with data used by the game. So for shipping I'd use binary files.

@gmfCoding

21 күн бұрын

As a gamer and programmer, I've always disliked binary configs for singleplayer, if I want to play the game with a modifued config I should be allowed to, if want to modify my game to give myself ten thousand items/coins, I should be allowed to do that, regardless of the outcome even if it means gaming breaking bugs, I messed with fire and I get burnt. Multiplayer is a different story.

@herrdingenz6295

20 күн бұрын

@@gmfCoding NO, you should NOT be allowed to unless the developer/publisher allows it. If you really need unlimited coins/health whatever - then write your own trainer/crack just like i did back in the good old 90s by analysing runtime code in assembler :P

@gmfCoding

20 күн бұрын

@@herrdingenz6295 well that's my point then, more publishers should allow it, they certainly don't have to recommend it or at the very least say "if you do this you risk loss or corruption of save data". This is in argument against your original opinion that: "just to prevent users from screwing around" IMO it's in the right of the player to screw around as long as it doesn't come to the detriment of the people's real health or others personal property. :) If the developer has other reasons for using binary files such as for performance reasons or development complexity then it may be excused.

@vaijns

20 күн бұрын

​@@herrdingenz6295 Thing is, if you don't give some people the option to do that or make it harder that could mean the game is less enjoyable for them and maybe they're just not gonna play it. E.g. The Sims is one of those games for me where I wouldn't have played it at all it if there weren't cheats because it allowed me to do what I enjoyed doing in that game (building houses). Would I have played it if I instead had to change a config? Probably, that's easy enough. Would I have played it if I had to install mods? Most likely not, having to download and install the mod loader and the mod itself (before ever doing anything I like in the game) would be annoying enough for me to just not play it. Would I have played it if I had to write a mod myself? Definitely not. Just the way players changing configs have to deal with any bugs or whatever that come with this, devs have to deal with players not enjoying the game or aspects of it, if they limit what players can do. Especially if it's just as simple as making a config file a text file instead of a binary, this freedom might be worth it. So yeah, whatever both sides decide to do, so be it but it might come at a cost.

Nice Peruvian sweater!

Pro-tip: Have the best of both worlds by making an editor that can directly visualize and modify the binary format.

i can open binary in notepad, i just can't edit it

My brain ( him not me): IF I HEAR THE WORD THUMBNAIL I'M GONNA BE LIKE 🙎🙆🙆

High level: Ez "Low" level languages for humans: too hard

2k lines will slow down a couple of nanoseconds, not worth to optimize it at all

me using base64 encoded textures so I can use git without lfs 😎

I code in binary

@eoussama

21 күн бұрын

I code by manually manipulating the ions inside of my CPU using magnets

@vaijns

20 күн бұрын

I code by making the world/earth itself my game and manipulating people

On modern computers I think the difference between binary and text for a 2500 line file should be in the order of milliseconds. We're talking about kilobytes of data.

@bits360wastaken

20 күн бұрын

In the order of milliseconds is incredibly slow for asset loading

@tedchirvasiu

20 күн бұрын

@@bits360wastaken Can you observe a 2ms difference?

@jamesmcmanus

20 күн бұрын

@@tedchirvasiu Sometimes, depending on when and how it happens in relation to other things, and how often it happens.

@vaijns

20 күн бұрын

@@jamesmcmanus I think the "how often it happens" is the most important thing. If it only happens at initialization it's fine, if it happens during the game regularly it's a problem.

@tedchirvasiu

20 күн бұрын

@@jamesmcmanus we are talking about files which would make sense to be human readable / editable: scene files, prefab files, material files. All of these would probably sum up to a few megabytes for a decently large scene. The big files such as mesh files, textures (images), audio files would never be edited by a human in notepad because nobody would adjust the position of a vertex in a 100k triangle mesh like that. And nobody would care to see how those vertices changed by looking at git diffs. And when I said milliseconds I was extremely conservative, we're likely talking about microseconds. I highly doubt parsing files measuring a few hunderds of Kb would amount to anything in comparision to other tasks.

Imagine Serializing into c++ file...

serialisation with a 'z' is a big crime

@flflflflflfl

21 күн бұрын

Oh no, guess I'm going to jail

@sonuaryan5287

21 күн бұрын

Me too going to jail 😢

@bigblackchode9628

21 күн бұрын

both are correct

@kotofyt

21 күн бұрын

programming war crimes, yay

@Redfrog1011

21 күн бұрын

Says who?

and Unreal Engine still uses stupid binary serialization for all its assets. as for me, the biggest disadvantage of it.

@zanagi

21 күн бұрын

tried to learn unreal using my c++ knowledge but the compile time ruins my whole motivation... Not forget to mention that it crashes when the components are not loaded properly, and has two different method of compiling 💀

@zdspider6778

20 күн бұрын

But encryption, tho. 🤔 In Blueprints you can use Ctrl+A to select all the nodes, Ctrl+C to copy, which copies it to the clipboard so you can send it to someone else (with Ctrl+V), in text format. I think it works with other node-based assets, too, like materials. But sending the .asset files themselves is a... philosophical decision that Epic made a long time ago to prevent people from stealing in-game assets.

please make more videos on how to use a hex editor. please