Tracking DAU and MAU in high-traffic sites is a challenging task. This article describes how we solved this problem at Logto.
There's no denying that as an identity provider, Logto processes a mountain of user data daily. One of the critical metrics we can't afford to overlook is 'Daily Active Users' (DAU) - it gives us a feel of how engaged our users are. But when it comes to tracking DAUs and Monthly Active Users (MAU), things can get messy. Managing massive volumes of data without affecting user experience is not easy, but it's something we've learned to tackle efficiently.
In this read, we're going to take this chance to walk you through DAU related concepts, explain how DAU, MAU, and a monthly engagement rate can be calculated.
DAUs are unique users who interact with your platform on any given day. These interactions could be anything - from just logging in, posting or liking something, or any other specific action that signifies activity. Tracking DAUs lets us gauge user engagement on a daily basis and get a sense of our active user base's size.
MAU, on the other hand, is an indicator of unique users who engage with your platform at least once within a month. Just like DAUs, active users can range from anyone who has simply logged in to someone who has interacted with the platform in some way.
When it comes to understanding just how often users interact with our platform in a month, we find immense value in a particular metric - the Monthly Engagement Rate (MER). This is not merely a number for us, but a pivotal element that heavily influences our pricing strategy.
MER reveals the frequency of a user's engagement by pinpointing the count of unique days a user logs in or interacts with our platform within a month. This insight isn't just useful; it's a key differentiator for us. Unlike the typical approach that solely focuses on whether a user visited within a month, we dive deeper, we strive to understand 'how often.' This 'how often' enables us to gauge the depth of a user's interaction with our platform and provides us with tangible metrics to deliver more value to our users.
In Logto, MER is a significant factor of usage-based pricing. By anchoring our pricing to MER, we ensure our charges align with the user's level of engagement; you could say we've turned fairness into a metric we can measure. While others might count a user who's logged in only once the same as a daily user, we differentiate, pricing based on the actual usage and value provided. Through this, we ensure a more transparent, fair, and ultimately, a standout approach toward pricing.
Usually, DAU and MAU are calculated directly from user logs. But as these logs pile up, the process becomes inefficient and presents challenges like data scaling and log retention. Not to mention, keeping records of user logs forever isn't practical due to privacy and security regulations.
That's why we came up with the concept of a DAU table. This table mainly includes two columns - user_id and date. The date can be set to the start of one day.
We have two ways to keep this table updated. The first one is a daily automated job that sifts through the user logs from the previous day. The second one triggers an update whenever a user activity takes place.
Whenever we come across a unique user_id, we add a new entry to the DAU table, which includes the user_id and the date. This is done using a SQL "upsert" operation which requires a unique key of user_id and date.
As a result, tracking DAU and MAU just becomes a matter of counting rows in the table. And the monthly engagement rate calculation can be done by combining group by and count.
The DAU table, which essentially is a condensed version of user logs, has dramatically streamlined our DAU and MAU calculation process. Not only did it make tracking user engagement much easier, but it also reduced the load on our system significantly.
In our journey of optimizing DAU and MAU calculations, we've always laid heavy emphasis on user privacy and legal compliance. We built our strategy around ensuring the DAU table wouldn't store sensitive user information. Instead, we just store anonymous identifiers (the hashed value of the normal user_id ) and dates of activity, strictly adhering to all relevant data protection regulations.
We would highly recommend this privacy-oriented approach to others in the digital service industry. Regardless of the scale or volume of user data, privacy should always get top priority.
In the end, data tracking doesn't have to be complicated, even for high-traffic web applications. A stable and efficient strategy like ours can make tracking vital user engagement metrics a cakewalk.
We hope that sharing our experience can inspire other digital platforms and provide ideas on how to develop innovative solutions for effective and efficient data tracking.