Recommended for you algorithm Q&A

[00:00:05][music] [music] [music] [music] [music] [music] [music] [music] [music] >> Mhm.

[00:02:03]>> [music] [music] [music] [music] [music] >> Start and then introduce.

[00:03:15]Should we go live?

[00:03:16]Okay.

[00:03:18]Everyone, I'm Sandeep. I'm a product lead on the discovery and recommendation algorithm. I'm excited to share like this webinar and Q&A with like Ben who's our VP of engineering.

[00:03:28]>> Hi, Ben here.

[00:03:31]>> And also we have Patty who's a product manager on discovery.

[00:03:34]>> Hi everyone.

[00:03:37]>> All right.

[00:03:38]We'll get started.

[00:03:41]So want to start with like how our recommendation algorithm works and how do we select a set of games to show to every user personalized from millions of games that are available on Roblox.

[00:03:56]Recommended for you algorithm works in two stages.

[00:03:59]The first stage we call retrieval where of millions of experience that are eligible to be shown to every single user on the platform, we pick a select set of games in the first stage. And here the first stage comes from engagement on the game from any source. This could be like ads on Roblox. Could be getting featured on curation or standout games are it could come from search, friends. It could also come from off-platform sources that could be Discord, direct links to our experience detail page, and any other sources. So, any game that gets a very small number of play from any of these sources are considered to be shown in recommended for you algorithm.

[00:04:49]Now, the next stage is the ranking stage.

[00:04:52]Of all the eligible games that clear the retrieval stage are ranked based on a set of signals which we share on the creator analytics dashboard called the home recommendation signals.

[00:05:05]These signals are counted only when a user clicks and plays from recommended for you sort and the recommended for you organic section.

[00:05:16]And based on how a user engages in the subsequent day, session, and like weeks, we factor in signals that includes play time, play days, engagement, co-play, monetization together to rank which game is the most relevant game for every single user, and then how we construct the personalized home page.

[00:05:40]Next slide.

[00:05:43]We have three key updates with this algorithm change we shared in the dev forum post yesterday.

[00:05:52]The first one is we've heard a lot of concerns on QPDR being very important metric and like drives everything in Roblox algorithm. This is something we're replacing and introducing play through rate and first play bounce rate.

[00:06:07]Essentially, we want to understand what happens immediately after a player or user clicks and joins a game. Are they playing at least for a few seconds?

[00:06:16]That's PTR. And if they're playing for a few seconds and leaving the game right away, we capture that information through our bounce rates.

[00:06:25]This is first indicator that like in the first session we showed a relevant game, the user tried it but probably decided hey, this game is not for me. I need to leave. So we capture that signal.

[00:06:38]Next one.

[00:06:39]However, we want to personalize the algorithm to the most relevant users. We hope that most users who select and play that game actually continue indulging and repeating engaging with that game. So here we measure the familiar signals that is play days, play time, qualified play sessions, intentional co-play days, spend days and Robux spend in different time periods.

[00:07:04]Did the user who engaged with this game came back on day one? Did they come back in the second day, third day, seven days, which is measured in the day two to seven signal?

[00:07:15]And then did they continue to engage in this game that we recommended to them in the 8 to 28 day period, which is in the first month or four weeks? And this measures across like the time windows gives us not only the shorter term signals, which we want to quickly capture and recommend this game to more users, but also understand the durability of how good a retentive game that you've built to give it to maximum distribution that more users will continue engaging and playing with this.

[00:07:47]And the third update is we're also trying to share the relative importance of this set of signals.

[00:07:56]We have, like I said, if users like engage in the game in the first session, which is play through rate and first play bounce rate, along with the play time and play days are the most important signals as a new game developer or a game developer who've already been building games on Roblox for a long time and those who have established games focus on the most important signals first. If you've got those signals locked down and working very well, then you can think of additional signals that can even increase your distribution of the impressions you get from home page.

[00:08:29]Here's the relative importance of signals.

[00:08:32]The first set of signals like of play through rate, first play bounce rate, which is actually a negative signal.

[00:08:37]That is if a user leaves your game after joining within 60 seconds and or within 60 to 180 seconds, we consider this to be not a great experience because they tried and they found some issues, they left, or they persisted for up to 3 minutes and probably didn't like it still, or the onboarding was confusing or the game what they needed to do was not very clear, or there was some crash and issues. We want to understand that why they're leaving and then like use that information to personalize it to better user who's in a better device or better suited for the game that we recommended to them. So, the first four signals will be the most important for you to focus on. Getting this right in the different time windows and periods is going to be most relevant and beneficial that shows users are engaging deeply and coming back every day or every few days in the four weeks. That tells us we recommended a really good game that users enjoy. And then you can focus, if those signals are really strong for your game, you can focus on the next set of signals, and all these signals are visible for you on the creator analytics page. And Patty will talk us through how this is shared.

[00:09:46]>> Hi everyone. Um there's already some changes live to your creator analytics page that you should be seeing today.

[00:09:52]So, we've backfilled this data for the last month, and initially you used to have six uh different charts here that were lining how well your game was performing on RFY. We've expanded this to 21, and what we are doing is we've leveraged the usage of different tabs so that you can have a overview of how you're doing across the different three time periods for each individual signal.

[00:10:16]We can also dive a little bit deeper into it. Um one other thing I wanted to make sure that we note is that as Sandeep mentioned, we now have a negative signal that you that captures how quickly users leave your game, essentially your bounce rate. So, do [snorts] make a note that as the higher the rate of uh of this signal, uh the worse it is for your game, essentially. You want to ensure that the bounce rate is lower, and that will be reflected in this section over here. So, at this time I would encourage all of you to go into your creator analytics page and go to the acquisition tab and start checking out the signals and how they reflect on your game so you can better understand the changes. And do keep in mind at the very top we also provide a link directly to documentation that allows you to reference uh the relative importance between the different signals and provide updates there in case they change over time.

[00:11:07]Great. Um with that being said, we do want to dive into some questions.

[00:11:12]>> Yeah.

[00:11:13]Bring them in. Yeah.

[00:11:15]>> Awesome. Um so, we're going to start with some common questions that we receive on our dev forum post, and then we'll dive into the Q&A.

[00:11:22]So, the first question that we have been getting is how are new games going to be evaluated by the algorithm, and if it's going to take 28 days until uh home recommendations and signals are available.

[00:11:35]>> Yeah. Uh so, it does not take 28 days until home recommendation signals are available.

[00:11:42]Like we talked about, the signals are broken down into first play in the first session, which is play-through rate and like bounce rate. And then we have day one signals of play time, play days, Robux spend, co-play. Then we have day two to seven.

[00:11:58]And then we have day day eight to 28.

[00:12:00]So, the reason for us to decompose these signals across different time periods is to not only capture the initial engagement and behavior of users who we recommend these games to, but also factor in longer term. So, the games in the first few days are captured in the day one signals and the PTR and bounce rate signals. And then as your game continues to do well, we have additional information to say like we need to distribute more and more impressions to this game as users keep retaining on the platform and the game.

[00:12:29]>> I think the best way to think about it is that as we have more data about you, we get more accurate at knowing who are the best users for you.

[00:12:37]And especially for complex games that actually users play for longer time, weeks and more, now they have the advantage as we learn more about that, we actually give you even more distribution. So, it is not hurting the first few days, it's actually going to still grow, but it gets to its full potential once we actually know more about that.

[00:12:57]>> Perfect. Thank you.

[00:12:59]Um another question that we get is I can see that there are three different charts for each metric. Are they considered separately for the algorithm with one type weighted more than the others? Or are they combined and the breakdown is just to help developers understand where improvement needs to be made?

[00:13:15]>> Yeah, they're all considered separately and then combined together to decide how many impressions your game should get.

[00:13:23]So, they are individually important. At the same time, you have also look at it collectively. It's very unlikely a game to have a very strong day 8 to 28 day signal if the day one and day two to seven signals are like very low or very poor.

[00:13:40]A user is unlikely to come back in the 28th day if they don't come back anywhere from day one to day 27. So, that's something you need to keep in mind. It's important to focus on them individually, but then collectively how the playtime, play days, PTR, and bounce rate are the most important signals is where I would start first.

[00:14:01]>> And um two things. One is as Sandeep mentioned, think about it as you need to retain an engagement on day one. You need them to bring them back on day two. And of course, it gets harder and harder. Your game needs to have deeper engagement with the users. That's the best way to think about it. That's why you need to consider all of them.

[00:14:20]The second thing is that please give us feedback. We're actually working on next versions of this. We know the current version has so many signals, but we're working on seeing how we can actually improve that view for you. So, stay tuned for us to iterate.

[00:14:33]>> Yeah. The goal for us to in the coming up versions of data analytics dashboard is to give a very clear actionable insight on what changes you can potentially make to increase the distribution of your game.

[00:14:48]>> Perfect.

[00:14:50]Thank you.

[00:14:51]One more question, which is how are new games evaluated in the Oh, this is the same one actually.

[00:14:57]What factors influence home recommendation impressions?

[00:15:01]>> Yeah, I think the factors that influence home recommendation are kind of fourfold. Apart from the recommendation signals, which we just shared about, which is what the changes discovery algorithm does, number one factor is how many total number of users on Roblox come to Roblox home page every day. That is changes by weekend versus weekday, changes in summer months when the schools are off and like end of sum end of the year during holidays, and is low when they return back to school. So, that is one factor. Second factor is the changes Roblox algorithm makes, which is what we shared about. The third factor is how does your competitor games, that is of all the games I can show to Ben or Patty, where does your game stand relative to that compared to all the signals? So, that is competition of for your game.

[00:15:54]That is the third factor, and the fourth factor is what you do and how do you change your game mechanics to improve the home recommendation signals. So, given these four factors, we will always share changes discovery algorithm makes very transparently and clearly to you, so that you are aware any changes this Roblox and discovery team makes that may influence the recommendation impressions, and you already know the changes you're making into your algorithm, that changes the your game, that changes the signals that go into this algorithm. The two factors that are kind of challenging for both us on Roblox discovery team, as well as you to know, is how the competitor game signals are changing, and how many new users are users who are eligible to show this game and how many users come to Roblox home every day, and that varies. So, those are the four factors, and that's where we're working towards giving more clarity into your analytics on what changes you need to make to influence the recommendation signals, even if you make the biggest updates to your game and drive all the signals to jump by 50%. However, all the other competitors of your game also make those updates and have their signals improve more than 50%, you may actually not see impression grow that much. So, that's that's the reason why I'm sharing all these factors.

[00:17:16]>> And one thing I know that a lot of you make tweaks to your game and basically the question behind this question is how do I know how my change impacted the impression?

[00:17:27]One good tool that creator team released was the experimentation.

[00:17:32]The nice thing is that you can actually directly measure AB the impact that you had on users that came in. You can actually measure the impact it had on the retention and the user engagement.

[00:17:44]So, the best way to think about the biggest lever that you have is improve your game, improve retention of your game, users engagement with the game and all of that. And the best tool for that is actually that creator analytics experimentation tool. Because that one zeros out your control has the same impact from competition, has the same impact of seasonality, but you're really truly evaluating the causal impact of the change that you made. So, highly encourage you using that tool.

[00:18:13]Right?

[00:18:13]>> Perfect. Now, we're going to start diving into some questions from the Q&A.

[00:18:19]Um let me start pulling them up.

[00:18:25]Perfect. Um so, we have our first question which says, "How does the system decide whether to give a game more impressions after its initial test? And can poor audience matching during that test prevent a good game from being tested again again?"

[00:18:41]>> Not really. Like so, the way these signals work is because it's kind of a moving window. So, if you have like a game update or changes, say you launch a game, you don't see day one retention being very strong, then you say, "Hey, maybe I need to fix onboarding. The users need to understand what's going on and let me update it."

[00:19:00]Given these signals are a moving window, if you launch it like say a month from now, the algorithm like shares this game to like a few set of users on home recommendations. When they click through and play that game, we continue to measure whether that user came back or not, and that's how it kind of always always trying to expand how many users your game should be shown to. And if the incremental users we show the game to do not retain or do not come back and do not engage, then it says, "Oh, maybe I got it wrong. This game is not relevant to Ben and maybe it's relevant to Patty." So, it will show it to Patty when she comes on the platform.

[00:19:36]So, that's how it learns and adapts. So, there is no cost or penalty for you to start with a game that didn't perform as well and then you make updates and changes to the game mechanics or onboarding or any aspects of that and publish a game and update and this continuous learning system will take care of any of the updates you need to make.

[00:19:56]>> Um two things I would add Basically, the core of what Sandeep is bringing up is that ML system are always both doing explore and exploit, right? And of course, in the beginning it's wider range because they don't have confidence of where they would land, but it's an always going on thing. So, if suddenly after a few weeks you make tweaks to your game and your game get better, yes, it will take a little bit, but it will actually go to explore again and will find the right audience for you. And part of it is because ML system always have a little bit of randomization, a little bit of exploration built into them. So, it's not like, "Okay, it makes a decision. This is it and that's it and you're you're screwed."

[00:20:33]>> And we want that because we want the game to reach more audience.

[00:20:36]>> design.

[00:20:37]>> Yeah.

[00:20:38]>> Perfect. We have another question. How will a new game the 28 data affect new release games? Is it going to take longer to get pushed by the algorithm?

[00:20:48]>> No, I think we covered this in our earlier question. So, it does not take 28 days for a new game to get picked up by the algorithm or shown to more users.

[00:20:55]>> Yeah.

[00:20:56]>> And the best way to think about it versus what was happening before is that if your game with a deeper engagement with the users, now once we understand that you have that, we'll actually give you even more distribution. That's the best way to think about it.

[00:21:11]>> Yeah.

[00:21:12]And can you two elaborate what D2D7 averages actually mean for play time and play days? It's a little bit ambiguous what it actually means and how it's calculated.

[00:21:22]>> Yeah. So, like think of like any time window averages as of all the users who came to your game on through recommended for you algorithm, what was their total play time and how many number of users who came.

[00:21:37]So, same ways for play days. It's like the number of days all the users who came to your game through recommended for you algorithm played that many days. So, in 2 to 7 days, they have 6 days and the average number of users play 4 days, 3 days, 2 days, 1 day. That's kind of like how the average gets decided on the number.

[00:21:59]>> And in practice, they're basically measuring two things. One of them is how regularly the user is coming in and enjoying your game. And the idea is that a lot of times games that get users to regularly play the game, they actually show interest in that game more. Of course, time, too. The reason why you have both is that they capture different pieces of the user intent.

[00:22:21]>> Perfect. Um can games with lower D28s still expect to receive traffic in their early life cycles or is the algorithm designed against games like this?

[00:22:32]>> Uh there's no design that says, "Hey, we shouldn't show this to more users." Even if you build like a game that's really good for deeply engaging and spend a lot of time only for a week or only for a few days.

[00:22:46]We want to have all kinds of games be successful on Roblox. So, it's obviously great to build very complex, novel games that we have consistently told you to build deeper, engaging, longer retentive games, but it does not mean like games which have shorter retention or like good engaging deep play one time play through story book games and all of those are not important. It purely depends on different users prefer different games.

[00:23:13]I for example may prefer a deeply engaging RPG, Patty may prefer a story book game, and Ben Am may prefer an Obby. So we want all different types of games to be successful on Roblox and also make sure that we personalize and recommend the most relevant user the most relevant game.

[00:23:30]>> I think the best way to think about it is that the learnings that we had from a lot of our experimentation is that we need both.

[00:23:38]We need amazing casual games because there's set of users that like it all even the same user, right? During the week they're waiting at a bus stop, they want to play casual game, or on weekend they want to play a deep game. We want to have both, but we want to have make sure algorithm assigns value to both of them and we felt that we are not assigning enough value for those deeper games before. So still if you're good at building casual games, good high quality casual game, there might be a game that you play for 3 hours and you're done with it. That's actually perfectly fine.

[00:24:09]We want those games in the platform.

[00:24:12]>> Perfect. Um we got a very interesting question. Um when updating a game that is already very old, at least 3 to 6 months old let's say with um around 50 CCU and the signals are increasing, is it still possible to start growing again?

[00:24:27]>> Absolutely. I think there is no reason like a old game cannot be revived.

[00:24:33]Uh like I said like there are two stages to our recommended for you algorithm.

[00:24:36]The retrieval stage considers a very low minimum play required for that game. I could have a dormant game that I haven't updated in like years and has no players, nobody's playing that game.

[00:24:48]Even the initial set of plays that comes from any source that could be ads, Discord, off platform sources TikTok, YouTube, you sending it through iMessage or a WhatsApp to your friends and family, all of those plays are considered for this game being featured on recommendation algorithm, and then the ranking stage takes over to say like we'll show it to a small set of users on the first few signals, and then continue to see how those users are retained.

[00:25:15]>> The best way to think about it is that every day our algo is testing your game with millions of users by showing it to them, by them then percentage of them playing it. The moment our algo sees that there's a difference, now when I show it more people take it, the algo starts showing it more. Uh the recommendation is working on a daily basis.

[00:25:36]>> Perfect.

[00:25:38]Um do PTR and first play bounce rate measure only new users acquired through recommendations, or do they also include returning users?

[00:25:46]>> This is only for users acquired from recommendations on their first session.

[00:25:51]>> Remember RFI in general recommended for you is only a lever to give you new games. Uh continuous play is reverse cron, like those other uh game uh uh sorts or distributions that we have, they're very different uh uh requirement.

[00:26:07]>> Perfect.

[00:26:08]Um are paid acquisition users, sponsored users, and organic RFI users weighted differently?

[00:26:16]>> I'm not sure I understand.

[00:26:18]>> So, uh recommended for you mostly focuses on recommended for you uh performance, right? Because um uh we basically show your game in recommended for you on home, and then we look at the performance of those games, right? Now, all the other sources of traffic you're getting is going to impact the global um distribution of your game, but as far as RFI is concerned for the personalization, it actually looks at I showed it to Sandeep, did Sandeep like it or not? Now, if Ban 'em comes through friends or that's not considered qualified because it's not a test that RF ID did.

[00:27:00]>> Perfect. Um how soon should we expect to start seeing games with higher PTR PTR get shown in the algorithm?

[00:27:08]>> So, PTR is one of the 20 plus signals.

[00:27:12]So, it's not like, "Hey, if you have a very high PTR, it solves everything." It does not. It only tells like user saw the impression of that game on home page and say like, "Ah, this is interesting. Let me click and try."

[00:27:27]That alone should not and is not telling that this game is loved by that user.

[00:27:33]So, that's why we look at additional signals. Did they How long did they stay after they click through and played? Did they come back after first day? Did they come back second and seventh day? Did they come back in the next month? So, all of those signals together tell us tell us how widely this game should be distributed and to which users are more importantly.

[00:27:52]>> PTR continues to be an important part of the formula uh because if you show your game to users and nobody gets excited to play it, then there's no way that you can actually get them to play your game.

[00:28:04]That's why it is important. But, I would say it's actually the enjoyment of the game in our new formulation is even more important than before. So, if anything, I think the value of users enjoying your game versus the PTR versus the evidence versus the creative is actually now this one is even more. So, the way I would think about it is that uh the high value actions are even more important. What do we call them?

[00:28:30]>> I think we're calling them recommendation signals. So, that's why we have it classified like the play through rate, bounce rate, play time, and play days are the most important because that clearly tells if you show this game on home page, will the user click and play through?

[00:28:45]Will they stay? And will they spend time, which us play time, and will they come back and have increasing [clears throat] play days? So, those are the factors. It's not just PTO.

[00:28:54]>> One of the things we observed in the previous formulation was that QPTR has too high of a weight. So, that's why you can see that we've deprecated QPTR.

[00:29:04]>> Perfect. How does the algo compare games by Does the algo compare games by genre, gameplay type, or experiences competing against the same signals and benchmarks?

[00:29:15]>> No, we do not use genre or gameplay type as an explicit factor in ranking. We make it very implicit by the signals that capture different styles of games and different behavior, and then we look at which users retained on which game, not necessarily by genre.

[00:29:34]>> Anything you want to add, Benam?

[00:29:35]>> If tomorrow you go randomly change your genre to something else, uh it's not going to be like, "Hey, I can hack it, and I'm in this genre I get more distribution." If that's the question, mind the question, because ultimately what we learn, if you're a good shooter game, is that we show you to users that like good shooter games, and if the algo sees that you perform well, it continues to show you to more shooter games. If they don't like your game, they will not see it. So, it doesn't matter what you tagged it there. It's one signal that goes there, but actually we have a lot more signals from user behavior and from your game that actually it doesn't play big factor. I think it's better to actually accurately put your category, because it helps with people when the biggest use of it to me is actually the evidence. When users want to choose your game, you don't want to um you want to be transparent so that they know what they're getting into. So, if anything, I would say think about putting the most accurate description, because this is what users use to decide to play your game. And if you put things that are inaccurate, then they actually bounce, and actually it's bad for you, because you brought a user in, and they're like, "Oh, this is not what I was looking for."

[00:30:43]>> So, to translate what I'm saying is, if I build a game and it's an Obby, but then put a shooter, a user who comes in may play through, but then like >> Yeah, they will bounce and they're like, "Hey, I thought it was a shooter game."

[00:30:54]Right? So, if anything, we need to make sure that we give the best information to the user. It will help your performance, actually.

[00:31:02]>> Yeah. And to add one more piece, if anything, having a very accurate video on your details page lets a user know is this game for for me as well.

[00:31:11]>> Yeah, that that clearly takes care into like whether user will expect when they play the game. That's what you have, so they hopefully don't bounce and then retain and engage.

[00:31:20]>> Exactly.

[00:31:21]Um did the previous algo measure D7 retention?

[00:31:26]>> We did not explicitly measure it as a D7 retention. We used a combination of 7-day signals.

[00:31:34]That's implicit into D7 retention, which is play days, frequency, play time, whole play days, spend days. All of them, if they are strong, D7 retention is expected to be strong. So, in some ways, it is captured implicitly, but not one signal saying D7 retention is the factor we look at.

[00:31:53]>> This is a bit of modeling complexity there. There's a reason why we said D1, D2 to 7, and D8 to 28. There is seasonality, meaning that there are users that might play every Sunday, things like that. So, you want to be careful a bit that defining as playing again exactly on day 7. So, it seem there's some details into it, but the best way to think about it is that did they get joy on the first day enough that they came back in the next few days? And D8 to 28 is trying to capture do they continue to get um value weeks later from the game?

[00:32:31]>> In a very similar note, we have a couple questions that are very similar to each other. And they're focusing on what the relative importance is between the different signals, and whether for example, whether PTR is more important than bounce rate, or D7 is more important than PTR.

[00:32:47]Uh maybe we can discuss that a little bit more.

[00:32:50]>> Yeah, uh it it's the way to think about it is like what Ben was saying earlier.

[00:32:55]You want users to pick your game to play when shown on home recommendation. That captures did they start playing? If they don't even start playing, it's not going to be great. So, the PTR becomes important. And if they start playing, did they leave right away because they didn't like it within like a few seconds or a few minutes? That tells, "Hey, like there was an expectation mismatch in what I thought this game is going to be and what it is. It's not for me." So, that's the bounce rate is captured. And then we have the set of signals on day one, day two to seven, and day eight to 28, which captures all those factors.

[00:33:27]So, think of it like a funnel. At the top, did users come in and play your game?

[00:33:33]Did they leave right away or did they stay? And did they keep coming back in the next few days? So, you can think of one in sequence of which one is the most important in that order if you want to, but at the same time, if I have if I build a game that's excellent in day eight to 28 day play time and play days, then it kind of implicitly ensures the users came and played the game. They also like came back on day one, day two to seven, and that's why they have coming back on day eight to 28 day as well. So, that's the way to think about it. Anything to add?

[00:34:06]>> And we have shared the table of importance, right? We have the I think PTR and then days and then play time and then play time we have. And the reason why days is important as and as Sandeep mentioned, think about this as repeat usage of a product, whatever product it is, is the best indication of user getting value from a product. So, PTR, which is our top one, is a user deciding to try that game. And then days active is the user deciding to continue playing that game going forward, which is one of the biggest signals. And now we have the order of them. But if your question is between D1, between D2 to 7, D8 to 28, the best way to think is sending pension that they all kind of a continuation of each other.

[00:34:50]>> And we have one that we sort of answered, but it came up with a few votes, and it's if it's a scenario, right? If we have 100,000 users joined 28 days ago, does the D8 20 to 28 average measure all 100,000 users? Essentially, if they only joined once 28 days ago, will it be measured?

[00:35:12]>> Yeah. Yeah. Yeah. That's the definition that all the users that tried your game, a month later, did they still like your game or not? That's the definition. And of course, it's average is 8 to 28, but that's the idea behind it. It averages of all the users.

[00:35:27]>> Yeah.

[00:35:27]>> And if you have zero users basically being around after a few days, it means that you're getting zero for that score.

[00:35:34]>> Yeah. If if those users don't come back after uh one session, then it's capturing the D1 essentially.

[00:35:40]>> Yeah.

[00:35:41]If everybody bounces within seconds, then D1 is zero, D2 7 is zero. If everybody just plays it for one day, then D2 to 7 is zero. If everybody leaves within a week, then D28 then D8 to 28 will be >> zero.

[00:35:56]>> And it's not that as we said before, we also try to predict it before even the user gets to D28. So it's not like, you know, we actually try to predict the user behaviors going forward.

[00:36:08]>> Perfect. Uh we got another question.

[00:36:11]Maybe I can tackle it as well. It's asking about the creator analytics, if we're going to be able to segment CTR and bounce rate by device, um and how they can understand their metrics a bit better. So right now we're in the process of designing D2.

[00:36:25]Um I'm actually meeting with a lot of you >> [laughter] >> to collect more feedback and understand what you want to see from the next version. What [snorts] I would encourage you is to provide provide feedback and comments on the different posts and there's also feature requests on the forum. I'm actually going through all of them and reading them out to understand what it is that you guys need. Um, so just stay posted for that.

[00:36:47]>> Yeah, and our goal with this would be to make sure you understand what signals are weak and where do you need to focus on whether it's region device or something else with the recommendation algorithm trying to find the best and most relevant users for your game who actually deeply engage while at the same time exploring new audience who may find your game interesting and see how they perform.

[00:37:10]>> Um, we have one with four up votes which is they just want to confirm day one equals the day they first played.

[00:37:18]Trying to understand what that means versus D0 which is something they used to look at as well.

[00:37:23]>> Yeah, I think like D1 is the first day they played.

[00:37:28]Yeah.

[00:37:29]>> I think we might have didn't done a switcheroo of definition. Sorry, we should have been more explicit about it.

[00:37:34]Seems like before we called it D0 now we're calling it >> No, we didn't ever call it D0.

[00:37:38]>> Okay.

[00:37:38]>> Yeah, we we had seven days.

[00:37:41]>> Oh, you're right. Yeah, so we didn't have >> Um, let's see. Maybe we can wrap it up with one last question. It looks like we answered most of them. Um, I'm looking through them.

[00:37:56]So, this one is interesting uh, with a lot of up votes. Of the millions of games retrieved, how large is the set of games that makes it into the pre-ranking list? Is the list short enough that we should be concerned about the metrics to qualify for?

[00:38:08]>> So, the pre-ranking list is for every individual user. So, Patty may have a very different pre-ranking list of games than Benam and I may have a completely different. So, it's not necessary that you have to be concerned about it. It changes for every single user. So, we're trying to consider all of the millions of games for every single user, but then the most relevant set of games that's important for Patty may be different from Benam and for me. And then we always explore, like as Benam mentioned earlier, we tried to show maybe like we should introduce this game to Patty and see where this ranks amongst those signal for Patty.

[00:38:45]>> That is slide that Sandeep mentioned visualizes this the best way. Thank you mother, the user comes to home page. We start from list of all of our games and we try to narrow it down to the final, you know, 100 games that gets shown on the page. In the stages we try to kind of narrow it down. In every stage we try to calculate the value for the user and that value is personalized. So, for every single user we start from considering all the games and we narrow it down to the final 100.

[00:39:15]>> And I've also seen sometimes question is like, oh, this home page showing recommendation showing this to wrong set of users. And the goal here is to not because we want the user to click and play through and continue engage. So, we try to find the most relevant game for this user and that's how we narrow it down.

[00:39:34]>> Perfect.

[00:39:35]Um, I think we can wrap up.

[00:39:38]Thank you everyone. Thank you for joining.

[00:39:40]>> Yeah, thanks a lot.

[00:39:41]>> so much.

[00:39:42]>> Yeah, thanks a lot for your questions.

[00:39:44]And I appreciate it. Hope this was helpful.

[00:39:47]>> Bye everyone.

[00:40:42]>> Mhm.