Development Update with Server Meshing Technology - 11AM PT Wednesday, July 3, 2024
Ойындар
This month, we shared in-depth details about our server meshing technology in Ashes of Creation, along with studio updates, and more! 👇 For Timestamps click "Show More"
Chapters
0:00 Intro
1:48 Reminders
5:56 Server Meshing Technology
1:28:52 Testing News & Answer Server Meshing Tech Questions
1:48:04 Studio Update
1:53:55 Art Update
1:57:17 Outro, Q&A - Didn't hear your question in our Q&A? Check to see if it was answered over on our forums: forums.ashesofcreation.com/di...
🔔Reminder that Ashes of Creation is an open and transparent development project. As such, you will see work-in-progress art, systems, and mechanics. Leave us your thoughts on our progress in the video comments!
💬 If you have any feedback on what was shown, please share it with us over on our Forums!
- For Server Meshing Technology Feedback forums.ashesofcreation.com/di...
Follow us on
👉 / ashesofcreation
👉 / ashesofcreation
👉 / ashesofcreation
👉 / ashesofcreation
🗣 Join the discussion by connecting to our official discord / discord
📺 Watch live the last Friday of every month at / ashesofcreation
Пікірлер: 303
Glorious Ashes of Creation Community, a HUGE thank you for continually tuning into our development updates. We are getting SOOOOO CLOSE to A2 coming your way! Our June showcase was the most technically oriented we’ve ever done, and it’s all in an effort to share with our glorious audience what exactly goes into making this MMO, and what is being tested in A2. We know these kinds of updates aren’t as exciting for some folks as the gameplay-focused ones, but they are important. We would love your feedback, and we plan on announcing the A2 start date at the end of this month! Also, we plan to begin sending out test email invites to some A1 users this month as noted on the stream, so stay tuned for more information!
@AszkalonOfVerra
18 күн бұрын
We all assume dear Steven is now better again. 😎 Thanks very much for (finally ^.^) another Art Update. I myself am really addicted to see anything new from your awesome Fantasy Universe your MMO Game plays in. 🙂
@rodney3297
17 күн бұрын
What a great job taking such a super techy topic and making it digestible to the masses. As a IT professional I can certainly tell you non tech people cannot entertain it for very long lol. Nothing can make your mother bored faster than taking about a motherboard.
@josephcruz8797
17 күн бұрын
I wish you guys were on the East Coast
@chutch1122
17 күн бұрын
As a software engineer, I have put a lot of thought into how I might design a dynamic gridding solution for multiplayer games (like MMOs). This stream was very interesting as well as reaffirming for me because the ways you are achieving scale are quite similar to my designs. I love the technical deep dive content like this every once in awhile. Keep it coming!
@milesnapue
17 күн бұрын
1
Stop saying it wasn't exciting because it was tech. We gamer nerds loooove this stuff! Stream was definitely a 10/10 💪😎
@AshesofCreation
18 күн бұрын
Awwwwww, thanks. Glad you thought it was exciting. We just know it's not everyone's cup of tea. ;)
@leanture3685
17 күн бұрын
@@AshesofCreation it is mine! :D
@darklink911
17 күн бұрын
@@AshesofCreation I'd say this was top 3 best dev streams and I think even for non-nerds it might have been more interesting than you think. I would say that apart from the deep dive into network engineering which the team and the presentation did a great job explaining in layman terms it shows something we all MMO gamers know. And that is laggy servers because a world boss is being killed or an event is starting somewhere in the world. What makes this dev stream so exciting apart from knowing how things work in the background is your problem solving, your goals and how you've taken on the challenges to solve them. You are not satisified creating a game with laggy server when world events happen like every other game has settled on and every MMO-gamer just has gotten used to without liking it. And you are pushing boundaries and tweaking the unreal engine to multi-threading just to get that effeciency to where you need it to be for your OWN standards and goals. That is VERY exciting to see because that means soo much more in the long run. It means you deeply care about delivering and improving the experience to something not seen and experienced before. Even after the game has released we you choose to play the game will see those improvements over the years to come after release if you keep the same attitude forward. That is what this stream also told us if you read between the lines.
@indianakawa3752
17 күн бұрын
@@AshesofCreation Players who have played DAOC or Warhammer Online know how important this video is. Good gameplay without server/graphics stability is not good gameplay. Thank you for sharing.
All of you did a great job breaking down and explaining the most nerdy tech ever. Bravo for the awesome visual aids as well. Excited for good performance in large battles!
@AshesofCreation
18 күн бұрын
D'aww, thanks! 😊
This is great content! An MMO that isn't built as a distributed system and can't scale horizontally will always be gimped and immersion breaking. What you guys are doing is paving the way as the standard for all MMOs.
@AshesofCreation
18 күн бұрын
💖
As a game dev doing both front end and back end stuff, I find videos like these very interesting. Good job
I might get flak for this, but it’s the honest truth. As a technically minded person, I haven’t bought into any of the ashes of creation “hype” until this video. This sold me. The sheer amount of innovation this is bringing to the table has made me a believer in this project. More of these please!
For what it's worth, this is actually a much more interesting topic to me than a lot of the standard content updates. My team is also building applications on Unreal Engine.
I appreciate the troll, Steve N. Great stream. I'm a tech nerd so this was fun!
@AshesofCreation
18 күн бұрын
Haha! =P All in good friend homie.
Was watching this whit my friend we had fun and learned something new
@AshesofCreation
18 күн бұрын
🧠
Absolutely loved the in depth technical discussion! Didnt understand much but loved listening to the difficult challenges the team overcame and did what sounded to be impossible! Keep breaking boundaries team!!! Also Im extremely excited to be able to purchase the A2 key cause im not a broke university student now! Hoping it'll be reasonably priced!
@AshesofCreation
18 күн бұрын
YAY! Glad you enjoyed it!
Coldest line from this video "I did some research on multithreading on unreal and everyone said it's impossible..... but we did it anyways"
I'm really happy to see how much everyone wants to improve the game to be at its best. Great job hiring these invested professionals and awesome job done by them.
This was bloody AMAZING!! One of the best streams honestly, loved it ⭐️
@AshesofCreation
18 күн бұрын
Aw, so glad you enjoyed it! Thanks for the kind words and following along with our development journey.
I loved this update! Amazing solutions. You guys should really patent and license these solutions out to others. It should fix the majority of the online gaming worlds problems. Standing ovation and Hats off to your developers!!!
Really loved the deep dive into server meshing, thanks!
While I know Ashes is focused on a PvP. I wonder if we will see large scale PvE events. it be nice to see an MMO where the undead or ork horde is actually a threat to the world and not just waiting in a field to be farmed for XP.
As someone that has recently started to learn about computer science, this was amazing stream. Thank you guys so much! Very inspiring!
This is incredible! Server meshing is one of the primary issues Star Citizen is working to implement as well. I'm glad to see this fantastic technical deep dive; clearly great talent at AoC!
Absolutely fantastic update! Major props to the team for executing such a well done presentation, as well as making some really heavy technical info palatable for the core audience. Round of applause, excited to hear more about the bard and A2 date at the end of the month 👏🏼
I remember some of the general ambitious ideas how to solve UE limitations mentioned on one of the very first videos and I thought you just abandoned this as it turned out to be too difficult, but man, here we are. I'm impressed.
Amazing work! I can't wait to experience the final product!
I really enjoyed the Tech discussion, even with little actual knowledge its really cool to take a look under the hood and admire the mass of moving parts and shed light on issues you guys are facing. It's been very informative. Thanks!
@AshesofCreation
18 күн бұрын
You're welcome. Thank YOU for joining and following along with our development journey.
Absolutely phenomenal video, I don't understand the complex parts, but I get how insanely complex it is to do. If it were easy, everyone else would be doing it. "But it's still not enough" hang onto that mantra
Thank you guys so much for your work. The work you guys are doing is what the gaming world NEEDS to move forward. I cant wait to help out in any way I can. Even if it is just on the forums. I also have a friend that will likely apply so I will send them a message today! also personally I LOVED the more in depth tech side of this. This is literally what I spend my freetime reading up on and tinkering with and its how I made my career what it is. This is literally the best video I have seen for this kind of tech and explaining how it all works!
@MargaretKrohn of alllllll people, you were the one who helped me get a PS2 Beta key. I conversed with you all those years ago... Seeing you here makes me happy that you are continuing to prosper. I look forward to seeing AoC getting more work done!
awesome, great job! super excited about all of this!
WOW. That's crazy! I love the new techs you've created. We can finally get back “massive” in ”MMORPG”. I can't wait for this! Thank you for explaining it, I love it!
Really enjoyed this type of content on the stream, amazing job!
@AshesofCreation
18 күн бұрын
Thanks for tuning in!
To be honest, I'm always more interested in the underlying technology than the game itself. So I really appreciate the deep dive.
Great presentation! I think I understood everything, haha
@AshesofCreation
18 күн бұрын
Yay!!!
Honestly loved hearing the detailed explanation!
This is so exciting to know more about the in-depth processes, so don't say it's not interesting. It's awesome! Thank you guys! Waiting this game so much!
Great update. Very interesting to dive into the tech behind the game
Having played Ark and ASA I know that multithreaded replication is HUGE!!! Also very nice job on that slides!
I can't wait to play this game love you guys being so open about development.
This was great, as someone that has worked in similar technology spaces (albeit not in game development) this was very cool to see
To whomever is handling the media on this channel, don't look down on the metrics- all this information is vital and needed for the community. Viewership will boom when the game is released.
This is massive! Great job guys, hoping for a timely release now with minimal bugs!
Just to give you some idea of public interest in server meshing discussion, I’m not an AoC backer or player and I probably won’t ever be but I’m here because of your work in server meshing. AoC and Star Citizen are the only games with static meshing AND working on dynamic meshing and there is a segment of gamers like me who have played games for a long time under the limitations of traditional MMO server architecture who are excited about any games being developed that are attempting to push past the limitations. That should give you some idea of the appeal of tech heavy discussions like this even outside your community. We know what you’re doing will seep out into other games in the years to come so we’re all rooting for you!
Tech streams (and art >.> ) are my favorite!
so excited to play this game in 10 years when its finally out
watched it twice. That's how good this showcase was. Thanks intrepide! GREAT WORK!
I'm a complete networking nerd, this was awesome. I run a 10gb home network that's turned into a chore with the new SSH vulnerabilities. LOL I'm just biding my time now. Now it's Ramp Time for "Bardic Lore" and "Stand-Alone Character Creator"!!! I may or may not get into spot testing, but I didn't get to test as much as I would have liked in A1, so it would be no surprise if I didn't make the cut before A2 comes anyhow.
@AshesofCreation
18 күн бұрын
💕
This was an awesome video. Best insight video I've watched on any game dev so far.
Great overview of the server meshing. The effort you are taking to ensure stability of the game is exciting. I have one thought though. At 29:35 your slide says that a server is responsible for one grid component. If memory serves this is how EVE did their system, and in massive battles load was still an issue. Assuming you do not dynamically subdivide as load increases, it may be wise to see if you can scale the containers your dedicated server system runs on and allow for multiple servers/processes for a grid. Keeping those in sync via gRPC or similar technologies wouldn't be that difficult in comparison to what you have already described. With a true no layer system in place you may have thousands, or tens of thousands, of players in an area battling(if history teaches us anything it is that players don't really think about server load when they plan massive PvP battles). I LOVE that y'all are talking about how you plan on doing your backend!
I loved this stream! You guys are doing awesome work and I am in awe of the dedication and passion you guys are pouring into this game.
We need more tech stuff like this. This was gorgeous.
I’m super excited for this game, as a comp sci guy from the era of c++, make sure your deconstructor methods are clean so you avoid memory leaks when players jump from server to server 1️⃣ idea would be to have transmitters to turn proxy copies to auth copies, and vice versa and know which properties need added or sent to server cleanup, very cool tech
I'm a CS student, on my free time I have this little hobby game project I'm working on and it's a MMORPG, of course, it's a very simple one but still, having to build the dedicated server and learning all about things like rollbacks, sending deltas, serializing, load distribution and how everything connects is super insteresting. So I LOVE having in depth reviews of the networking, there is very very little content since this is a very complex topic but for those inside of it it's amazing to look at.
Awesome, Thanks for the in depth Update
Amazing video guys. I've been tuning in here and there for Ashes, but this showcase of how you're pushing the networking boundaries for MMOs is inspirational. Definitely makes me curious in providing feedback or even applying
I really liked this tech showcase its intressteding to see what U r doing behinde the curtains and getting Infos on stuff a Player dosn't See in His normal gameplay
You guys are doing a stellar job! Making magic happen! I'm super excited to see how this will work in game and the effects it will have for players. In cities around main npcs and shops is going to be the main test though, you definitely want that big social side of things.
very cool breakdown. I can't wait for this game to come out.
Wait, wait, wait ... ok, so in your display of a server going down @52:48 ... it doesn't only effect that server but every surrounding server that was in communication with it. Data is lost without a replication layer because no entity survived that has the current state within that server as well as all data that was to be communicated while the server went down. What happens at the authority borders between servers while it is down? Does it become an invisible zone wall until a new server takes over the other sides authority? The server that went down has all the entity data under its authority, so does that mean it can be the only server that can spin up to take that authority slot without major data loss?
@Hat_Uncle
18 күн бұрын
nice, I thought of that as well. They must have a "backup service" server for each Data Server, to pull from on spin up. If the crash and spin are reasonably fast then the other servers would show some minor glitching because there would be no transactions until spin up. So, if you were a player attacking something across a boundary, it would appear to do nothing? That's my thought anyhow. They cannot have you attacking the fake without response after all, that's too pure a cheese. I LOVE this Technical stuff!
@d12duke
18 күн бұрын
Deltas will occur I'm sure but it's not hard to recover db information running HA. I don't know for certain but a child authority will likely be used. Lots of ways to prevent data delta.
It's good to be an informed player. This video was great info.
@AshesofCreation
18 күн бұрын
💕
Thailand people are waiting to play ❤
Keep up the hard work us players cant wait to make it home. It will be super exciting to see the largest battles and fun game play ever created.
Hey! Great choice with microservice architecture for your MMO RPG project. It’s an ideal fit! As a developer who loves optimizing service performance, I can really appreciate the noticeable impact of your work. Although my contributions are usually less visible, I can see the impressive results of your efforts. I’ve always wanted to work with gRPC, and it seems like you’re using it for cross-server RPC calls. Your implementation of load balancing, server scaling and proxying is remarkable-you’ve practically built k8s for gaming. Kudos to you guys!
I loved this, I haven't been following these updates very much - but I might considering following a lot more if it we get techy updates like this
This was an amazing stream. Best one by far in my opinion. This really helps to appreciate the hard work going into this game and the future is off the charts bright. Thank you for this.
I recall a game called Rift and it was all cloud based server side. One dev talked about seeing 1300 people in one zone, no issues. That was pretty awesome.
1️⃣ thing to know is I love this technical deep dive
It was a great stream, really missed these more technical updates and I'd love to see many more like it. Can't wait to start testing and see you all ingame! edit: Also, as a braver of worlds backer I'm perfectly fine with you selling more alpha keys. If it can help you stress test the server go for it, the more the merrier and more cash for you to work without pressure.
Loved this update much more than usual. They are always great, but the tech is more interesting. I'd rather see this stuff than the usual "gameplay" stuff that what I think would spoil it for me when I do start playing. I'd like to experience that when the time comes rather than knowing what's coming. Amazing work peeps 👍🏻👌🏻🎉😁🔥
For those unfamiliar with software development, here's some insides: Story points - unit of effort measure for a certain development task Backlog - it is the bag of all development related tasks that are yet to be done Debug - it is the way of checking the code as it runs, or register (log) all what is happening Server meshing - it is the mechanism of server interoperability which is the way they speak with each other to create a working server cluster Milestone - it is a set of things that once done fulfill the milestone and normally creates a patch or a minor/major version to be tested out or deployed Concurrence - entities competing each other for resources such as players asking for server responses Expensive in development - how much computing and hardware resources a certain thing consumes
This stream was FIYA!!!!!!!
Wow, that slide deck is art/science on its own! Love so much the team is sharing this work. Thank you all! 💖
The complexity of the tech used in mmos is probably one of the reasons why there's so few good mmos. Good job explaining this stuff.
Can´t wait to play this
Ill be honest, i am a developer for internal systems for businesses (automation etc.) but listening to this type of problem to solve... i think im in the wrong field. This sounds fascinating to sink your teeth in. I dont think i live close enough to work but just the sheer passion is heard!
My dream, a true massive multiplayer sandbox, great job!!!
Did they say how big each zone is (give or take)?
We've evolved into Ashes of Citizen!
I don't understand network coding but because of my job I like to make things more efficient as an applied mathematician. I really liked the tech behind the scenes. I have seen similar attempts at this in other MMOs, most of which were abject failures. I have a feeling as a layman that things will be different here, you are going in the right direction, this will be huge step. Glad, I can support this project as a backer.
as a gamer and a indie game dev i love these. these things are exciting :))
About server meshing, replication problem. I am not very familiar with tech behind Unreal but, is it possible or reliable to calculate part of the relevancy checks on client? Or gives some sort of info to help server out. Here is an approach, client has registry table which gets filled up on first load. This registry table holds the current surrounding data, which you already provide. This table also has version hash or something like that. Like version controlling softwares, this hash indicates the version. Therefore, server checks clients latest values and only sends changed ones. If hash does not appear in the log, server need to send all the data as it did in first load. I dont know how possible or effective it will be. Wrote it down in 2 mins. But main idea is involving client to do some work so server load maybe lessen.
I didn’t understand everything but it was really interesting to listen to. Hopefully there will be KZreadrs that will break down on what exactly was achieved that wasn’t before and how this translates to players.
Very nice to see such an in-depth peek under the hood!! Thx... Being a techie, I do have (many) questions: does a client connect to a new server when crossing over a border? Or is there some other gateway in between? (would give a bit of extra delay)
As a nerd I approve this video thanks for the transparency ashes team you guys are changing the way videogames are developed.
@AshesofCreation
18 күн бұрын
💖
I love this. Great video.
Does these alpha2 new keys give access to betas as well?
So if a server is down or in maintenance at a particular grid. Would the grid where that server reside be inaccessible to players?
It was freaking amazing and inspiring! You should definitely bring this to GDC as a presentation, maybe it would attract some people to the team. Also great to hear Alpha 2 purchase coming back, I missed my chance last year but I'll be sure to get it now.
Was a nice live stream! We watched it live from Germany we went like oooh and ooooh and wow already 2 hours in was it supposed to be this long? :D
@AshesofCreation
18 күн бұрын
How's the weather in Germany? We love a good oohing and ahhing. ;)
so where can I buy access to the alpha2? AoC website shop doesn't show anything
@AshesofCreation
18 күн бұрын
It's not available yet as noted on the stream it will be available at a future date, we are just giving folks a heads up. ;)
@levonschaftin3676
18 күн бұрын
@@AshesofCreation good to know, thanks.
They are so confident that they allow people to buy packs again! That is wild!
@AshesofCreation
18 күн бұрын
WOOT!
@Hat_Uncle
18 күн бұрын
well, actually, only the Beta holder can Upgrade Packages. The other people can buy A2 Only keys for a reasonable price. They have to walk a fine line there, make sure the Package holders feel fine with it, keep the price as low as they can, but make it feel like a Value for new people. I figure, based on my A1 package, value and components, that $10-$20 will be reasonable.
I hate that I missed getting a copy of the game for testing. Really looking forward to playing one day.
@weirdafters4007
17 күн бұрын
no one is playing at the moment so you didn't miss anything lol
Can't wait till the start of Alpha 2 Might start making content over this and take a deeeeeeeep dive into the mechanics and seeing what is actually good and could use a tweak It has me hyped watching this game for the past 5 years slowly being crafted and made by people who actually care about the creation they are making. 1
This sounds cool. Also sounds like an easy way to shoot your self in the foot though. The trade off for compartmentalization or decentralization is always complexity, and complexity is also difficult to work with. I am praying for this to work cause it sounds great.
1:16:00 when you remove the foliage from the spatial grid what happens when I shoot an arrow at a bear running through the woods? does the pass through the trees?
@Hat_Uncle
18 күн бұрын
based on what they were saying, I'd GUESS that the server looks at where you are pointed and anticipates what needs to be there and puts them back as needed. Is that before or after you fire? Does it have to calculate trajectory first? OR does it wait for the "fire bow" Event? No idea.
@Reac2
17 күн бұрын
If it's a removable/interactable tree, it should let you shoot through it at all times anyways. This is an MMO ,not Tarkov
Question will the micro servers have PID over each other like will server A. Info happen Before server B like in runescape?
as a developer I love this stuff.
- Since you made multithreading work, have you tried to delegate work to multiple servers? (or is sending the data to process across servers too expensive?) - You were talking about hotfixing faulty servers: isn't having a system to detect and replace them automatically possible? (backup servers) - Is it possible to have a server-swap system ? (Having a bunch of monster-servers that takes authority over crowded areas (that are not usually crowded))
threading replication, don't the actors depend on eachother so how can you reliably control order? blocking thread?
The best development update I've seen ❤
As UE5.x progresses - does modifying aoc version of UE5 prevent these updates from happening more regularly?
Will we have an armour set that would display the Cross?
Is there a technical limitation that prevents 2 servers from handling the same area, but the actors are split evenly between the servers?
@bin27kk
16 күн бұрын
That seems to be what dynamic sharding does
i'll tell my grandkids to try this game when its finally comes out