A podcast about about building and growing app businesses. Brought to you by RevenueCat.
Listen anywhere!

Eric Seufert, Rovio

David Barnard
December 2, 2020
Listen to this episode anywhere!

In This Episode

Since Apple’s big announcement about upcoming privacy changes in iOS 14, mobile app developers have been scrambling to understand how these changes will affect their businesses. With IDFA effectively dead in the water, developers will no longer be able to use device-level attribution and high-resolution tracking to send targeted ads to their users. So what can they do?

In this week’s episode, we asked mobile marketing expert Eric Seufert for his take. Eric has had quite a career in mobile. From VP of user acquisition at Rovio to his recent consulting projects with subscription app companies, Eric has a depth and breadth of experience with mobile apps and games that few can match. He’s also a prolific writer. He wrote a book on freemium economics and has written hundreds of insightful articles on his site Mobile Dev Memo.

In this episode, you’ll hear about:

  • The difference between user acquisition and quantitative analysis in mobile marketing
  • What Apple’s privacy changes mean for the App Store economy
  • Why the post-IDFA world presents a huge opportunity for mobile app developers

Follow Us:

David Barnard: https://twitter.com/drbarnard

Jacob Eiting: https://twitter.com/jeiting

Eric Seufert: https://twitter.com/eric_seufert

Here’s the Outline of Our Interview with Eric:

[1:07] Managing user acquisition at Rovio; freemium app dynamics at scale.

[3:15] Incrementality testing; the power of the Angry Birds brand.

[5:32] The efficiency of the Rovio funnel; 20-30% click-through rates during the launch of Angry Birds 2.

[6:30] Why it’s hard to determine the effect of a specific ad channel; holdout testing.

[9:10] The difference between mobile and traditional advertising: response time; Pepsi Super Bowl ads.

[11:09] Quantitative analysis in mobile marketing; QuantMar.

[11:46] What is a UA practitioner? Performance marketing; understanding metrics on revenue, installs, and campaigns.

[13:42] Why you don’t want the platforms themselves monitoring your performance metrics.

[14:10] Google’s perverse incentive; can budget and bid increases improve campaign performance?

[17:00] Lack of IDFA gives more power to the platforms, less visibility for marketers purchasing ads.

[17:21] Google Analytics; monetization and transparency.

[19:54] Algorithmic trading; data modeling.

[22:10] Why Eric doesn’t like the term “user acquisition.”

[24:06] What Apple’s privacy changes mean for the next few months/years of mobile development; IDFA and SKAdNetwork.

[29:35] Deterministic vs. non-deterministic paradigm; ad optimization.

[31:00] Why post-IDFA is more work for advertisers but provides a bigger opportunity.

[31:47] Filtering engaged/monetizing users at the content level vs. the ad platform level.

[31:54] Probabilistic measurement; the updateConversionValue parameter.

[35:38] Personalization in the post-IDFA era; real-time changes to the product experience based on user interaction.

[37:43] How App Store changes have shaped the entire mobile app market; Top Charts.

[39:32] Was Apple’s IDFA decision an intentional strategy, or were they unaware of how much it would affect the mobile app development industry?

[40:00] Apple doesn’t appreciate the complexity of the App Store dynamics.

[42:16] Eric’s thoughts on Apple’s motivations: control over App Store distribution; Epic Games drama.

[42:50] Apple has identifying information on individual users; high- versus low-resolution tracking.

[44:25] Prediction: SKAdNetwork changes will most likely go into effect this January.

[45:10] The problem with conversion paths: the race to the middle; extreme price discrimination.

[48:55] Will smaller apps (with smaller ad budgets) be disadvantaged by SKAdNetwork? Calm; Headspace.

[50:16] Apple seems to be paying attention to developers and want to get this right.

[51:05] Mobile Dev Memo Investment Syndicate.


“When you’re spending a lot of money and you’re showing a person five ads a day, incrementality is critical in figuring out how much value did this particular ad contribute? … That’s the more interesting question because when these budgets get really large, you get these signals coming from all these different mediums, and figuring out which of those actually drives value is more important.” - Eric

“That’s what’s so interesting about the mobile ecosystem: that immediacy, that kind of lack of friction.” - Eric

“Just doing the fun clicking around Facebook Ad Manager, that’s not performance marketing. That’s advertising operations or something.” - Eric

“Google’s always going to tell you to spend more money. Whatever question you ask Google about improving your performance, it’s like, “Oh, just pay more money!” - David

“These systems were designed to sort of alleviate that need, on the part of the advertiser, to not have to have this big team of data scientists working on these models. Like, “Hey, we’ll do it for you!” And to be honest, Google could do it better than any individual advertiser could — it’s Google. And just the fact that they’re syndicating all that data across all these different advertisers, they just have more data than any single advertiser could. And that’s a good thing.” - Eric

“This is something I think Apple has failed at since the very beginning of the App Store: understanding the way their individual, seemingly small decisions end up shaping the entire market… Now we have SKAdNetwork, we have one ConversionValue, you can’t update it in the background, you can only do it once, it has these weird timers… the entire market for apps is going to reshape around the shape of SKAdNetwork versus it having been shaped around the existing tools.” - David

“There’s billions and billions of dollars being generated in the App Store, and it’s such a tiny little market, so people put in the effort to find ways to game, to maximum advantage, any point of leverage that Apple gives them. So now people are poring over the SKAdNetwork documentation just trying to find ways of like well, “How can I best use ConversionValue?” It will shape the design of apps. It’s going to be a totally new design paradigm.” - Eric

“I just get really scared because ultimately if [Apple gets] it wrong, we’re gonna see apps moving in a direction that just is for them to extract the most value across the ecosystem and not necessarily provide great experiences for each user. I think lower resolution tracking, in this sense, actually can lead to that.” - Jacob

Like this episode?

Subscribe to Sub Club on Spotify or Apple Podcasts to get the latest news on mobile subscription apps.

Episode Transcript

Hey, you're listening to the sub club podcast, a show dedicated to the best practices for building and growing subscription app businesses. We'll share insider secrets from the top subscription apps on the app stores. Let's get into the show. All right. Welcome to the sub club podcast. I'm David Bernard and my cohost is Jacob IDing.

Hi today. Our guest is Eric Siefert. Eric has had quite a career in mobile, from VP of user acquisition at Rovio to his recent consulting projects with subscription app companies. Eric has a depth and breadth of experience on mobile and games. If you can match, he's also a prolific writer. He wrote a book on freemium economics and has written hundreds of insightful articles on his site.

Mobile dev memo. Welcome Eric. Thanks for joining us. Thank you for having me. So typically on a podcast like this, we would go right into tell us more about your career and all that kind of stuff. But I just wanted to skip that they can read your LinkedIn or whatever, but there were two things that I thought were just especially interesting about your background.

And we'd love to just dive into those instead of giving a deeper overview. So you, at your time at, Rovio got to experience user acquisition at a scale that like few people in the industry ever get to experience. you must have had, quite a budget you were managing and millions of acquisitions.

So I'd love to hear just like any war stories or what it's like to manage you at that scale. And just tell us a little bit more about your experience with Rovio. Yeah. Sure.  the budget that we had at robe, you actually wasn't, large, at the time. So when I joined the company had never really done UA consistently before they had a UAT team, but, they were just transitioning into, like a freemium.

Mindset. and they were re-engineering the whole organization around that, around building freemium products and games as a service. So they had done well, they had done way before it was usually like out launch, So they would have a una kind of budget for the launch. And then they would have beets, maybe if they did, updates like big guts game updates that they wanted to promote, but it wasn't there, wasn't this kind of infrastructure, for doing like.

Scaled continuous UA. And so I came in and build that out, but I think what was interesting about working at Romeo was seeing a, that the power of a brand, right? The power of kind of a consumer brand. because, while  the amount of money we spent wasn't, it wasn't, noteworthy that just  the mileage that we got with that budget, what was right.

it lowers your CAC, right? When you're you have a really strong brand like that. Yeah. Like the brand just drove so much, so much sort of efficiency in the kind of ad conversion funnel. So Just higher percentage of click through rate, higher percentage of install rates.

People knew what they were getting now. Now what we didn't, I think do while I was there and which isn't a, which isn't to say that like I would approach it in the same way now, because the industry's changed a lot and evolved a lot since then. So this was like five years ago, but what we didn't do very well while I was there was to catch us for incrementality.

we were just, showing ads to people and counting every single kind of, click and install, and dollar spent subsequent to that install, from that user as like just totally incremental revenue or doing a whole lot of testing around, what would happen if we just didn't advertise to that person?

if they hadn't seen this ad, would they have installed the game anyway now? I don't think so because I just, part of that part of where incrementality becomes interesting is really the level of spend, what, which of the ads that you're showing. If somebody drove the revenue, not the question, like the sort of this sort of negative scenario.

In an incrementality test for most advertisers, isn't it? if I hadn't shown this ad with this user, just apropos of nothing having installed the app, probably not. without any sort of inducement, they probably wouldn't have done anything. Now, the question for the real sort of important question and really interesting question to ask is did this ad inspired them to install the app versus the three, other than three others that they saw this day?

So when you have a very low level of spending that mentality, isn't that important? and I think for us, I don't think it would have changed much, but how we were doing things back then. But,  when you're spending a lot of money and you're showing it a person like five ads in a day, the incrementality is really is   it's critical and figuring out okay, how much value did this particular ad.

Contribute versus just the other four or five or whatever, and if I hadn't shown them that one, but they had seen, the one on Facebook and the one on Twitter and the one on TV would they still have installed? that's the more interesting question, because when those budgets get really large, you just get signal coming from all these different kind of mediums and figuring out which of those actually drives.

Kind of the, which of those actually drives value, and contribute to value is more important. but anyways, it just,  we didn't really think about that much back then, but yeah, just seeing that the efficiency of the ad funnel was really eyeopening. Cause I,  I'd come from another company that did casual games, but  the funnel metrics were nowhere near what we saw at Rovio when we launched, actually some of the networks thought, they had, they had a, like an internal problem.

Cause the click rates are so high. Has there ever seen something like 20%, 30% click through rate, And there must be some issue. It must be like a, an analytics issue, but it wasn't, everybody clicked on the app.  if you have differentiation and brand is a great way to differentiate, really like angry birds, fun game, but it's a block knocking down game.

it's not anything like, there's a lot of those. But they had this brand and this would have been 2015 would have been posted, they had moved out of just mobile apps. At that stage, they, I think they had like their store and the movie was in the works. If it wasn't already.

And they had already been building this like media property around the birds.  so yeah. So I, to your question on incrementality, how does, like, how do you practical? I guess it's easy enough to do  a hold back test where you say Oh, show me the revenue for a cohort that I'm going to leave out of this like targeting campaign or something like this to see how much revenue they generate and then look at the difference.

But how would you go about measuring? Yeah. Some of that more difficult, less like more nuanced incrementality. So it just seems like really difficult to get, quantitative, information on, it is, but I think everybody is going to Everyone's going to be taking a masterclass in incrementality going forward.

that's ultimately where we had. I think ultimately we had is if you say we can no longer attribute the level device, then everything becomes an exercise in incrementality. that's measurement. so I think,  w we're all gonna be a lot more knowledgeable about it, but at the conceptual level, like incrementality, just based on, like observing variations, And ad spend and whatever your outcome metric is.

So if you think about, Hey, we want to index all of our, we want to, we wanna, we want to measure, ad spend on the performance of revenue. then the way you do that is you just look at, variations in that ad spend and variations revenue and look for this sort of  correlate Tory effects.

So if I think about a lot of people, one way to doing that is to like an engineer. A holdout. So I'm just going to send it, the only problem with that is you're creating this, you're creating this kind of environment that won't exist. when you're applying the learnings from that, right?

so if I am, I want to know what the incremental impact to Facebook's then essentially to cut Facebook, spend off and see what happens. okay. You're learning. What happens when Facebook is cut off, right? that's how these other channels, how these other channels behave when Facebook is cut off.

It's not necessarily telling you what Facebook contributes. because they're not the areas you're taking the partial derivative with each of these, but they're not independent variables. So each one of these networks, each one of these spin channels is interdependent. So you can't look at the individual effect.

I think that's always been, from a data science. that's the hard part of data science in general is like, how do you isolate action or cause and effect. And you mentioned, these like broad correlative studies, right? Like just see, do you see? Cause I feel like, especially when, speaking of Rovio, but eventually when you get to this scale and you're on so many touch points in so many channels, like I imagine a brand like Pepsi, they don't have a great idea of incrementality.

I can't imagine, Like they don't know how much their super bowl ad produces. no, I doubt it. I, so I know that some of these CPG brands do bring like some sophistication at this. they're not just like throwing money at agencies. and a lot of times they're, they're leaning on the agencies and there's a lot of agencies that are really sophisticated.

the only prompts, the only problem  with what they do though, is like, there's just even more noise because there's not a direct response. that's what makes VR kind of such an interesting field to work in is because you get that immediate response.  I know immediately if someone installed my app or not, or I can say  I know that if a user installed an app and it was from an ad.

Right then that's that probably happened like pretty quickly. there wasn't like a long sort of deliberation period on their part, Oh, should I download this? is this the right kind of half, but it's free, but he walks around like they do. When they're thinking about buying a new car, should I done that?

I don't know. Let me go take a walk on it. Yeah, exactly. there's no comparison period. And there's no   price shopping, like those are for free apps. And it's just yeah, just download it. Yeah. The switching cost is zero. and th there's like opportunity costs.

Of your time is pretty low, Download this app and play it for play for 10 seconds. So I think that's what that's, what's so interesting about the mobile ecosystem is that immediacy, it's that kind of lack of friction, even though there's some sort of frictional steps between seeing an ad and installing an app and spending money, but it's, they're pretty minimal, right?

It's not, it's like you said, something like buying a car, they're getting tighter and tighter, People are building more and more that time to value into their apps. Like subscription tools are getting closer to the beginning. I'll I'll I think. Largely for the measurement effect, right? It's the close of this will make it at fast.

You're gonna be able to understand these channels. Yeah. That was quite the rabbit hole. I do want to take a step back and this actually plays into exactly what you guys already were diving into and that's your a Q and a side quote, Mar and I love the name quant Mar. And so I was just curious,  obviously even in this.

preliminary part of our conversation. You talk like a quantitative analyst. So is, was that kind of your thinking of picking the name and then what do you think about as far as like quantitative analysis in the context of mobile marketing and like, how does that all fit together? I think it's basically the core of it, right?

I think that's a tool set that you need to have to be good at it. That's one thing like that started bugging me a while. Like a couple of years ago, it was just like, there was this kind of explosion in, demand for like you a practitioners, right? are you a person like we need to hire, are you a person?

and like that title. Just wasn't very evocative of what the job is. And actually like the job was, could look so different at, so at, depending on what company you worked at. So  I just hated that. I hated that saying like I'm a UAA manager or I run a UAA team. It's just, it didn't feel like it described what I did given that, there you, a manager might do something very, just a very different sort of like day to day workflow.

my sense is that just to be really good at, and I started moving away from even using the word, use the phrase, user acquisition towards just like performance marketing, Like my sense of is that being really good at performance marketing? It's  it's really understanding the measurement side of it.

Because I think,  that's really where,  the performance aspect of it is gleaned, right? Like just understanding how. How can I attribute revenue and installs to a campaign and understand the sort of underlying like yield,  efficiency like that.

That is about the model. It's about the model that you build that, th that kind of manages that measurement process. So like just doing the button, clicking around Facebook ads manager, that's not performance, right?  that's just whatever that's advertising, operations or something.

And  and then,  in the last year,   the efficiency of all this stuff, got absorbed into the platforms themselves, right? So they started doing a lot of the heavy lifting. but that's dangerous. You don't want the platforms doing your measurement, right?

that's actually not a good thing because they're always gonna privilege their own campaigns. And they also don't. Yeah. Good news, mr. Marketer, your campaigns are going well. It's like a joke in the UAA community that like, Google's always going to tell you to spend more money.

whatever question you ask about. I love that it's like, Oh, just pay more money. Because they, we analyze your campaign this week and it turns out a little bit more. And I think we're going to get you to your goals. It's such a, it's such a rip off. It's like the, it's like the quack medicine stuff too, It's Oh, you just need one more treatment.  Yeah. to some degree it's true. Because a lot of times that is like paradox that it can be. It can be true that if it's based on performing well, if you increase the budget or you increase the bid and then increase the budget, It could, and I've seen it happen, right?

There's a lot of things, especially with Google, because Google is  bucketed all of their different channels into this sort of UAC product. And so you might get stuck right on the sort of lower tiers of, in terms of just like traffic value, the glow in a lower value  traffic channels, You might be in Google display network. And that's where all your ads are being funneled into because your bid is not high enough. And it's actually not like graduating into the higher value. I like discreet break points where you'll break into like better. Yeah, exactly. And then with the more budget, it just allows more experimentation in those different, in the different buckets.

so if you just increased your bid, but given some constraint budget, all the budget's going to go to the one that's gotten the majority, the budget today because. because that's how these kinds of optimization algorithms work. So if you increase the budget, then it instantiates, it's like experimentation cycle.

And if you increase the bid, your bid signup, you can start experimenting with the higher value channels. Then you actually could increase your performance. But it is  yeah, that's, I don't think most like account reps. What articulated that way. And I don't think most UK managers interpreted that way.

And so it is and it's just funny. It's Oh, you're not performing well, increase the budget. it's okay. At some point you don't understand why that could be the case. And then also, like when do I know that's definitely not the case. it's actually just not a function of  the sort of like a constraint of my bid and my budget.

but yeah, you're right. that's like a funny it's it's a, well-known trope that gets trotted out, Oh, just increase your budget and pay for it. Every they've created such complicated systems, That they almost need to like. They can't expect every customer to have a quantitative marketer on their team or somebody who can do modeling, at all.

But so they, it makes sense that they would have to build some Oh, like here's how it's performing. I know it makes me wonder, what's the. what's the long tail of these platforms, revenue that comes from just like under optimized spend. so many times I've gotten like for like mom and pop shops and things like that were just like totally off the marks, like search terms and things like that.

They're just like hoovering up. it takes, it brings it all kinds of questions.  w this can go into Some of the IDFA stuff, but how the lack of idea phase actually just pushing more power into these platforms and less visibility to the people actually placing the, placing the buys, for ads.

But yeah, I've taken to every time I opened the, the Google analytics sheet, it's just dismissed. We have all this suggestions. To alleviate that need right on the part of the, on the part of the advertisers to not have to have this big team of data scientists working on and these models like, Hey, we'll do it for you.

And to be honest,  Google could do it better, any individual advertiser. Good. As Google. And anyway, you just, the fact that they're  they're like syndicating all that data across all the different advertisers. They just have more data than any single advertiser could.

Good thing. I think. and I think it's not even a bad thing that they're not that transparent about it. they could be more transparent about it. But the whole idea is that look, if we're transparent about it, then you're just going to try to tell us what to do.

Like we're telling you that w we know better what to do and the amount of money, and we're incentivized to do it really well, because then you'll spend more money with us. So I don't think there's any sort of like perverse incentive there. if Google delivers good results to you, then you'll spend more money with them.

The problem though is Google is only incentivized to deliver results that are just good enough. You tell it, Hey, I need to hit 110% ROAS on this budget. They're going to optimize towards that, right? Like they're, they have no incentive to deliver to you 120%. I new local, new Maxima. They'll get caught in some local maximum, and then just be like, Oh, that's good enough. Yeah, exactly. because, cause if the fact of the matter is like they can spend your money. If they're delivering 120% ROAS to you. That's just less money that you're spending right now. You could say, long-term, you're going to, you're going to increase your budget with them maybe, but maybe not now in the short term, if they're delivering 120% to you, why not just, redirect some of that budget to the sort of high output channels to the lower output channels, That are less competitive. Because everyone's going to continue bidding for that inventory that's high output, right? that's producing the high ROAS. Why not just take some of that budget and put it to the low output, like the low yield channels that are like underutilized and get you to exactly where you said you want it to be and you can't complain.

So my Target's 110% you've delivered to, and I have no idea because I don't have that visibility. I don't know if actually you're actually allocating like 20% of my budget to this channel is driving. No. income to me. Why is that? because I'm monetizing it, otherwise it would just be totally on you.

It doesn't matter. It doesn't matter how this actually affects your flywheel. They know the realities of your business.  it brings to light why quants exist in algorithmic trading, right? Like you're looking for an advantage because you're essentially bidding against other companies that are trying to more profitability.

Profitably act,  by these, this advert to these advertising channels. So there is an advantage, right? You're like searching for who can model this better, who can find a better fit. is that a correct way of thinking about it as terms of the quant analogy goes? Yeah. that's why I actually chose that,  that phrase too, because I think  if you think about, working at a company where you're spending, I don't know.

5 million a month. Which is a lot of companies do that. That's not that rare,  then you're managing a pretty reasonably sized portfolio of money. If you think about the turnover there, it's high.  and so  why shouldn't,  I think it's just a matter of the positioning, why, Why performance marketers and digital marketers, aren't making as much money as like you're essentially looking for a return.

Exactly. It's different because the capital is moving, which like changes the game a little bit. It's not sitting capital. It's like being reinvested constantly, but right, exactly. So if I spent 50 million in a year, presumably  some of that came back within the year and was reallocated, right?

So depending on the LTV timeline and the recoup timeline, maybe that was actually only like 30 million. It just got used, some of those dollars got used twice, but still regardless, if I deployed 30 million in a year,

And so like why, aren't, why aren't people that do performance marketing paid the same as people that are,   that are running, portfolios at hedge funds, Advocate for that as a company that doesn't employ any performance. I agree.

The scale is different, right?  even a 10, even if you 10 X to that, it's a $300 million fund is not that big. but still I had, my point was like, I don't think it's okay to think about. The problem is the sort of that performance marketing role. I think I got  defined by the sort of least technical people in that like broadly.

I think that the, sort of the idea of the qualifications. and therefore the compensation of that role got defined by like the lowest common denominator, which was the button pusher, right? The person who has a spreadsheet, they've got two monitors open, they've got on the left-hand, it's a spreadsheet, which says, Hey, I should bid this much on this campaign.

And on the right hand, it's Facebook ads manager and saying, okay, here's the box. And I'm just gonna push my mouse on there. I'm going to backspace. And I'm going to update it with what this thing says, user acquisition. That's why I don't like the term use requisition. and I thought like maybe reframing, it was just better in general for the field.

Because if I'm just kinda thinking about this, from taking a long-term view on it, like you want to attract the best people to it. You want people to  feel like the qualifications are,  are pretty, sort of August, right? Like they're pretty impressive. and you want to feel like, Hey, the value add is very high.

And so it starts with just the name. The naming conventions don't call it like UAA manager or Facebook advertisers. They call it quant quantitative marketer. And that, that just changes the perception. I met my, the same thing might happen to happen with data science as a role in the last five years where it went from, your business analyst or something like this, somebody just Abloh and now it became this very attractive field.

And we're so early in this, You're talking about five years ago at Rovio being very, that was a long time ago. It was five years. It's not that long ago.  so I have a feeling that, yeah, if I were to bed, I think you're, I think you're pushing the branding in the right direction.


So we're already like 20 minutes into this and haven't even really gotten into the meat of what I wanted to talk to you about. So let's, let's jump to the future. so you know, we've hashed and rehashed. The IDFA changes and everything that's going on there. So we're not gonna,  if you're not familiar with this and you're listening to this podcast, you might be listening to the wrong podcast, but go read blog posts and other things about Apple's privacy changes.

So we're not gonna rehash that, but what I want to hear from your perspective, especially taking this more quantitative approach to performance marketing, Is what do you see these next? So step one step back, Apple did just last week, push off these changes at, we think like six months ish, they said early next year.

so from today in September, 2020, we likely have six months ish until Apple does enact these rules. And then once they're enacted. things are going to take months to shake out and really understand,  how things are going to go. So I would love to hear from you your thoughts on what the next six months look like, building up the tooling, that's going to be effective in this new paradigm.

And then what your longterm view of mobile marketing is in this new, privacy friendly paradigm that Apple has a shared in by edict. Yeah. so I think that, there was always going to be like a three-stage rollout to, of the kind of post IDFA, infrastructure and strategy. So I the metaphor I use for this is it's a tornado just obliterated your house. Like you've built this house and you enjoy living there. probably that Hey, I live in tornado alley. It's only a matter of time. Built it on Google or Apple street, right?

Yeah.  but Hey, it's you got comfortable there. And so you never really felt super motivated to, I don't know, to protect yourself or move. I dunno, the metaphor does extend that far, but anyway, and it's tornado cons and it just obliterates your house. it totally, or let's say it's you see it barreling down the cornfields and, it's going to obliterate your house.

step one is,  is, is you find a tent. Like you need something to live in  to shelter you like post obliteration, You just need something that's functional and keeps you dry. and  you find a tent and then, you, in order to build on that,  in a way that's like incremental,  and additive, but it's also functional, cause at the same time, like you're going to have to give you get the tents, you got the house, you got the shelter, you're able to shelter.

You're able to keep the rain off of you or whatever. What does that mean in this context? That means, I've got, connectivity. to my, to,  to Apple via the SK network functionality. So I've instrumented the SCAD network, functionality in my app. I've got the ATT setup. And if I'm, if I choose to do that, and I've got the update conversion value being fired, In whichever way, I'm choosing to do that either I'm choosing to go with the MNP solution, which is to just upgrade their STKs and my app and have them do that. or I'm doing that myself, but probably in this, like the 10th version of that, the sort of just pure shelter version of that. Isn't taking all of the existing events that I already have in my app.

And I'm just instrumenting the update conversion value with that. I'm not building new events. I can't, I don't have time. this was September 15th. I didn't have time to do any of that. I'm just instrumenting update conversion value with existing events. And I'm using those as a basis for predicting  LTV.

So that's the kind of tent version. Now. Now I have a dream house that I want to start designing. But while I'm living in my tent, it's actually really hard to build my dream house.  So what I'm going to do is I'm gonna sorta upgrade the tent and I'm gonna, maybe it'll turn into a shack.

It I'll put some tin roofing on top of it or something and install,  a flush toilet or something. And so   that,  manifest in the kind of. IDFA  environment as  I'm not just using the existing events that I already had instrumented in my app and firing up the conversion on those.

But actually what I'm doing is I'm thinking about, okay, how does someone  evoke their LTV to me, it's not on any given event, but it's actually like chains of events. So like I had a workshop that I was giving, and I was talking about this in the workshop. It's Hey, I did this and I did this and I did this.

And that's different than if I just did this, but I didn't do that second thing. And I did that. So it's that's different than if I just did this one thing and not the other two. There's all these different combinations of things that you could do and figuring out like, which of those combinations.

is really, is really like indicative of actual value and figuring out what the values are for those. So that's the kind of shack version of that. That's like I've got running water. And I sleep on a bed and not before, but this isn't my dream house. And then all the while I'm designing my dream house and inspecting it out.

And the dream house is like the long sort of terms, state that we're moving to. The thing that everyone was always too comfortable to build. Which was moving completely away from deterministic measurement and building a sort of non-deterministic measurement paradigm that actually is even better than it's deterministic, Because I'm leaning completely on the sort of end product. A filtering system in product classification system that before I was just outsourcing to Facebook and Google, right? So like what Facebook and Google got really good at. Over the last few years was a routing the people to your app that would monetize at the highest possible degree, right?

To the highest possible degree. Like the people that would monetize the absolute best. They got really good at routing them to you. and they did that on the basis of receiving every single event from within your app and from within other people's apps, just ingesting all that data. And classifying these users on those, on that basis, putting them into these subgroups, like basically taking all the monetizers and all the engagers and breaking them up into subgroups on the basis of shared characteristics and then testing your testing, different ad creatives, if yours, against those subgroups to optimize the ad spend, and then sending you this, like pre-vetted, pre-qualified.

A group of people now that's really great, but that doesn't scale beyond what it is, right? Facebook and Google are going to send you the amount of people they're going to send you. And you can't really scale it beyond that. You can just try to build your monetization in such a way that just extracts more money from them, right?

That was ad optimization. now you can't do that. So what you're forced to do is to actually take the harder, approach, but potentially more rewarding, which is take this big diverse group of people that Facebook and Google and be sending you. Try to classify them yourself to try to give everybody every single one of those people, what they want, like with, with some potential, a degree of monetization attached to that level  of receptiveness.

And so if you do that, then you have probably more of an opportunity than if you just depend entirely on Facebook, filtering out, monetizer filtering out. non monetizers and sending you just that highly targeted group of people. I think the opportunity here is bigger, but it's more work is incumbent on the advertiser and it all that filtering is going to happen.

Now it's the content layer, whereas before it happened at the ad platform, So when you say probabilistic measurement, you're, when Apple Google's now sending you a much less targeted, much less,  like you said, not just these monetizers, but like a larger group. so now in the world is you get an ID, the IDFA and you can measure exactly how much that person spends.

we can, revenue cat can also send that to attach value to that offline as well. and now With update conversion value. And I think that the biggest thing is update conversion value is a one shot. They can't get, you can't keep updating it. You can't keep measuring it. what, I guess I'm having a hard time as like visualizing, like what do you actually have?

what do you actually, do you have this like broad unclassified group of users? what do you actually know when you're saying probabilistic? You're saying like, Oh, this is like the LTV distribution. Is that, what is that? What. Is that what you're trying to arrive at and then informing that back to the advertisers or w what are the, on the ground?

what are the actual, steps you're taking to move towards probabilistic? So you're exactly right. So that's irony. That's  kind of 2016. Facebook introduced ado 2017, they introduced VO 2016, Google went black box with UAC. So all this, all that did was, it was to allow advertisers to outsource all that work.

But.  the work being like the manual button pushing and the changing the bids and all that kind of stuff, but what then advertisers had to do, in order to optimize those campaigns was to like, actually move deep into the funnel of the, the app experience and to find those events that were the best sort of indicators of.

Of ultimate value for that user. And to send those back and say, Hey Google, here's finding more people that do this, right? Because this is what's and then try to move as early up these experiences as possible. So that the turnaround time, and that was really quick. this is just an amplified exercise in that, because before you know you, you really have one event where they yo you had one event, right?

This is what I'm optimizing for. But now,  it's just a more extreme version of that because, yeah, you really only have one event. You only have one event that you can even pick from. And also that, you have to construct that in such a way that like,  it can't be that single event.

because if it's just that single event, then people like go through the experience and they trigger that event. And then they trigger something later that has a higher conversion value that you've attached to it. You won't actually know she's got a trick you have to give to construct.

The experience is such that any time you see a conversion value, that you know that the cascade of lower conversion values is implied. And then you have to optimize for that one thing in a way that like, it either happens or doesn't, and it doesn't happen that sends a signal of like very low value.

So the idea here is that all of that work that like UAA teams were basically becoming like analytics teams anyway, because all the button pushing was absorbed into the, to the Facebook and the Google platforms. now that's just becoming an even more extreme version of that because you only get that one conversion of all the value.

That you want to capture has to be distilled into that one conversion value. And like not only is it an exercise in analytics, it's an exercise of product design and these two things are coming together in such a way that like the UAA team really is. They have, like strong influence on the product design, especially in the early stage.

And they're basically doing like this sort of,  analytical exercise in determining like, how can I, not only can I, how can I measure the value of this, chain of events, but how can I construct the chain of events? Such that it signals a lot of value. So how can I put the user on a path?

and then I think like the biggest kind of skill upgrade going into this, like post IDFA area era is like the personalization, right? So how can I, in real time change the product based on how the user's interacting with it, to  put them on a path that gives them the opportunity to engage to whatever degree they want.

So not have that singular product experience, but actually had this product experience that sort of like changes with their interaction that gives them more of what they want. And unlocks that value for them. As I see that, Oh, they want this. Let me give them that. As, since you're no longer just targeting this one profile like high spenders, I no longer have to have one product experience.

That, that is used to find one niche group of audience. I can have all the product experiences and cause I'm getting all the audiences, So I should. Yeah, the thing you hit on there, this is going to be David and I have talked about a lot, which is this notion that I can't help, but think that this peculiar behavior I'll describe it as peculiar for the update conversion value in SK ad network.

Is going to feed back into the shape of the weight. it's one of those things where you're like you go to a city and like, why is it shaped this way? You're like, Oh, because there used to be a river here, Or something like that. So there's like cascading effects that are going to change how it's especially.

not all apps, but apps that are built on monetization or a user acquisition are going to have to build around basically this SD, this API, which I don't think was super, fully thought out. And And I, these are the externalities that Apple doesn't consider. I'm not sure that they realize the like amount of like leverage and data analysis and all of the work that's going in to making apps monetize.

I think Apple is just let's just let them set a value and it'll be fine. It's no people are going to go nuts. It's a competitive market. And people are going to try to find advantages, right? This is something I think Apple has failed at since the very beginning of the app store is understanding the way there.

Individual seemingly small decisions end up shaping the entire market. It's like when Apple was,  from the very beginning we had the top charts and the top charts were totally based on volume. There was no concept of grossing or whatever. And so when an app went on sale, it doubled tripled, quadrupled the number of units.

So that pushed it up in the charts. And when you got high in the charts, That got you. Exposure. So you could get attention just by dropping your price and going on sale. Oh. And then dumping a bunch in UAA too, right? Like just like doing everything you could just to try to break into that top 10 and then, and then it's self-sustaining because you're hiding the charts.

It gives, like it gives,  social all because of one tab. In the apps tab on the app store and and there's been so many things along the line that I, that Apple has done that shaped the entire app store market.  that. That they don't seem to have a clear crasp on how much it's actually shaping things.

And this is like that next level example. It's like now we have ASCAP network, we have one conversion value. You can update it in the background. You can only do it once. It has these weird timers and there's so much going on. And like now for the next five years, or, we'll see if they make changes, but I think Jacob's exactly right.

It's  the entire market for apps is going to reshape around the shape of SK ad network versus it having been shaped around the existing tool. So I think that's going to be really fascinating to see that play out. So Eric, do you have, are you gonna, do you prognosticate about Apple's strategy here?

Do you think they're playing 10 dimensional, chess by  see how they, the industry would react or I, you don't have to reveal your sources obviously, but  I'm halfway between, they did this on purpose to try and get people to move, which fine, whatever. And then half of me thinks like they, they realized halfway through Oh, we stepped in it.

this is probably a bigger thing than we expected. Yeah, I think they just, they don't appreciate the complexity of the apps or dynamics. It's like you said, if it's billions and billions of dollars being generated the assets, that's a tiny little market. And so people put in the effort to find ways to game.

To maximum advantage, at any point of leverage that Apple gives them. And so now people are pouring over this scandal or, documentation, just fight, trying to find ways of   how can I best use the conversion value and you're right. it will shape the design of apps.

Like it's gonna, it's gonna be a totally new design paradigm, Because you're trying to get that high value conversion value to trigger early. And you're trying to map that to something that happens really early. So it's going to concentrate a lot of monetization really early on. which I don't know.

it,  it's going to be, it can be jarring. But it's just, it's going to be the only way you can really scale growth, is getting that signal as fast as possible. and getting it as clearly as possible. So I don't know, but I think, like I said, the motivation for this is they don't really, they don't appreciate how  complex the ad.

The sort of, the performance marketing ecosystem is on mobile. They didn't really fully comprehend.  how, how  how frail it is, right? if you remove this piece, like it's not, we're not just able to adapt around it, right? Like you break it, you break it, you buy it, like you should have to replace it right.

With something that's workable. Now I think, tackles credit, I think provides a lot. Now of course, it's not as complete if I'm operating on the same paradigm as I've been operating today. Yeah. it's not complete, I can't do device level attribution with it, but that was the whole point.

I think the whole point is now. Okay. We have to shift, we have to shift our approach to this sort of like data science centric. thing where we're actually doing more instrumentation in the app. We had that one moment and so let's get everything we can. And at one moment they basically just didn't hit a bomb and I don't think they were ready to have to deal with the sort of fallout.

but I think my sense is the motivation for this was, it was more at the, at broader. A tech ecosystem layer, right? So I want to hurt Facebook. I want to take back more control over app store distribution, right? I don't want ads being what determines which apps are popular.

I want to be the person that determines which apps are popular and that with the fight, with Epic, with the fight with, Hey, they want to have total absolute control over the content and the app store. And this is just them.  pushing further along on that path. I, what I don't understand is okay, fine.

We're going to let Apple federate all the tracking, right? So Apple has individual identification on these users. Of course they do. and we're just all gonna go fine. Let's let Apple do that. they're trusted consumers, trust them, developers. We don't have a choice. And so be it, what I understand with specifically with update conversion value is let us just send high resolution data to you.

You can claim that, scramble it. You can do whatever you want to anonymize that don't give us any promise that you're gonna pass that straight through to networks. but let us have at least like the resolution and so much so that. like going before we don't have these, you don't have these wild, every funnel looks like it's being trained, trying to be stuffed into a seven bit number or whatever.

just for that. So I don't know, my hope is that maybe they've been listening. They're probably listening to his podcast. Hello, Apple.  but they'll take some of this. Cause I do think that there are, we've written a much as much about this, but there's some really good benefits to this system and I don't think anybody's opposed to it in principle, but yeah, I just get really scared because ultimately.

If they get it wrong, we're going to see apps moving in a direction that just is for them to extract the most value across the ecosystem and not necessarily provide great experiences for each user. And I think lower resolution tracking in this sense actually can lead to that. So I don't know.

That's my hope, but we'll see. We'll see if they I'm guessing they're going to time it with their release in March would be my guess just on timing. I'm planning for this to happen. January. I feel like that's, I think you've got to stay as disciplined with your sort of response plan as you were pre announced pre delay.

I think it does. I think what a lot of teams are going to do is going to relax, and then it's Oh shit, it's in January. We're like,  we just had Christmas, blah, blah, blah. Like we're not ready. We're not ready. And I don't think Apple games a second freebie. There's no second stay.

Of execution  it's, there's going to be zero sympathy, I think the second time. But I think, you're just gonna see a lot of teams that just didn't get anything accomplished between now and whenever, the announcements. But I think there's a problem with the conversion path thing is okay.

So if we move, we moved into this system where. Everybody moves to the middle. Everybody. We're not, we're no longer doing like this really broad spectrum of different types of apps that are designed for a single kind of a niche audience. and that monetizes them really well because Hey, this niche audience wants this specific use cases.

They're willing to pay a lot for it because that's just what they want. but rather we build these sort of like containers that could be any kind of experience that, we dream up for these big broad audiences. Yeah, that could move you in a dark place because you start talking about like hardcore, like extreme price discrimination.

you start trying to, you start trying to like dynamically adjust everything, to try to like, push people onto the path that best express their intents and monetize. And, you get like these monster, like    there's just these Frankenstein's monsters apps that have   10 different people could end up playing 10 different games or 10 different people can end up experiencing you're building specific systems, just to try to get people to monetize as much as possible, almost like a casino.

I think  that can move things in that dark direction.  cause once again, like it's, like you said, it's like when people are adapting and it's the sort of first order, the first order adaptation, and then there's all the ways that adaptation evolves down the line and affects users, lots of users who never saw an ad.

but if we think about  Hey, I got to personalize because I have SDI network insight, implement some personalization and then I just. Over time, improve and improve and improve that. And then it turns into the system where you, Hey, I've got a system that,  I know I can extract a thousand dollars from a player in the first 30 minutes.

Just cause I've tuned it, to respond to this, To respond to their sort of behavior in such a way that it's like super efficient at it. And that is almost like an unimaginable consequences of this. But if you think about that's, what's going to happen if you start telling, if you start motivating devs to start focusing on hardcore personalization, that's exactly what's going to, you're going to end up with, you're going to end up with the assistance after a couple of years, like the first response is just Hey, do you want the $8 starter pack?

Or do you want the $2 starter pack? That feels pretty benign. But the sort of mutated evolved version of that is Hey, someone, like I've got someone coming in the game and I've got a system design that can monetize them $2,000 to 20 minutes.  that's where this heads and they're not just, they're not the people that are like the most obsessed with this type of game.

cause I think that's  there are games that do that, but you say it's a niche audience. These are people there's like a hundred thousand people in the world that like these kinds of games and we're finally bringing it to them and they've always wanted, that's different.

That's a little bit different than when you say, Hey, we built this brig big, broad tent and we welcome anyone in. And then we just like separate people on the basis of something that they don't even really know about themselves, potentially the tools that you utilize to get to that point, are not just product features.

They're like psychological features, right? Wow. we've gone way over on time to end it on that cheery dystopian note about the future of the app store. Yeah. So I was going to say, so we've got four minutes left. Let's do one more quick hit, give me a 92nd answer so we can end on something at least a little more interesting and positive, hopefully.

one of my biggest concerns, and this is going to be hard to answer in 90 seconds. But you can do it, Eric. one of my biggest concerns with this shift task network is this smaller apps are going to be disadvantaged. So if you're spending, 10 K a month and trying to grow your app, where do you see things going from, for the smaller apps versus  a million dollars a month, a huge company, who's already a calmer Headspace.

That's got a big team and. Can spend the money to experiment. Where do you see this going for smaller ops and being able to do the incrementality measurement and probabilistic and all that kind of stuff? I think the incrementality stuff, again, I don't think it becomes super relevant at a super low level of scale, Because you're not trying to discern between which ad provided the value or how much value to these. Add provide, because you're not showing people four or five ads a day, You're just, you're not spending as much. I don't know. I think my worry is that we're smaller advertisers. you're going to get caught in this privacy threshold trap, where you have to have a certain number of conversion values that get sent before Apple starts reporting them.

You have to have a certain number of installs from a campaign before Apple search reporting them. And so you could get, if you're diversifying your traffic too much, which a lot of companies do. If they're spending like a hundred K a month, they're still using like a decent number of networks.

They just won't get any data back.  geez, that wasn't the a positive note I was hoping to end on, but, we'll see how this all plays out. And the great thing is,   it is, it's an economy and economies adjust, and Apple has a lot of levers that they can pull between now and the announcement to improve things.

And then once it's released, there's a lot of levers that they can pull to make adjustments over time. And I think them delaying the changes is actually at least a positive sign that they're paying attention and want to get this right. so maybe that's our positive. Is that as dire as benevolent monopoly?

Yeah. As a, as Doris something seem, I do think Apple's paying attention. And I think that, there's levers for them to pull, to keep improving things over time. but yeah, I think we need to wrap up, Eric real quick. Where can people find you? And if anything, last minute you wanna, share.

Yeah. find me on Twitter at Eric under underscore super,  find me on the web@mobiledevmemo.com. Just today. And now it's the mobile dev memo, investment syndicate. so launching a syndicate out of the community, should hopefully be some cool investments that were announced pretty soon.

yeah. And that's it. that's that's where I can be found. Thanks so much, Eric. It was great having you on. Thank you. Yeah. Cheers. Thank you to make sure you never miss an episode. Subscribe to the show and your favorite podcast player. Thanks so much for listening. Until next time.