Pinterest and Annie Ta

Since 2019, Annie Ta has led a cross-functional effort to build more inclusive products across Pinterest (she started at Pinterest in 2012 and has held multiple positions across many different teams). Her team’s mission is to “build inclusive experiences that empower everyone to create a life they love.” A year ago, following the murder of George Floyd, the company stepped up their efforts to ensure they were building inclusive experiences into their product.

Since its formation, her team has made major improvements to the team’s foundation feature: skin tone ranges, which enable users to filter certain beauty-related searches to show content that is more similar to their own skin tone.

In our conversation, Annie talks about:

Annie Ta: Pinterest started building inclusive products back in 2018 with our foundational feature called skin tone ranges, but it was just a small project within a larger team. In 2019, we formed a team that was focused exclusively on building inclusive products. At the time, that team's mission wasn't fully realized , but the goal was to figure out how we can get enough resources to build more inclusive product experiences. Last year, in 2020, we formalized the team that I lead today. Our mission is to build inclusive experiences that empower everyone to create a life they love. This builds off of Pinterest's overall company mission to help people create a life they love. But we really emphasize everyone because we want to make sure that no matter who you are and how you identify that baseline of finding inspiration is equal for you so that you can find that inspiration.

AT: The interesting thing about building inclusive products at Pinterest is that we are a relatively large company. We're a public company, but we're not a huge company with around 2,000 employees. I always think of us as being big enough to really make a difference in the industry, but small enough to be a little nimble. We’ve thought deeply about how inclusive product experiences can contribute to the overall business and user experience of people who use Pinterest, known as Pinners. To begin, we decided that the initial measures of success would be simply creating experiences that we hope put BIPOC at the center of our decision making when it comes to product experiences that we build. After we launched these experiences, we learned that it's about adoption of the experiences, as well as, awareness of them and making sure that we can determine whether or not these experiences are contributing to a better version of Pinterest.

AT: One of the big things that we learned is people simply don't know our skin tone feature exists. The reason why is that it only exists on a limited number of product surfaces. To begin with, we turned on the feature for a small set of queries related to beauty searches. So as people use Pinterest to search for something like makeup our feature would appear. But if you're not searching for things like makeup, you may not know that it exists.

In the back half of last year, our team focused on growing awareness of the feature and helping people understand that it existed. We used a number of channels to do that including press, in-product marketing and off-platform marketing. It was the first time in my experience as a product manager that our marketing campaigns greatly contributed to the adoption and usage of the feature. I found that really fascinating. I've worked with product marketing teams a million times and it wasn't as apparent to me for other features that I've launched as it was for this feature. We saw a huge jump in usage during our marketing campaigns. In a lot of cases, for me as an inclusive product PM has been about building cross-functional relationships across teams to make sure that we're doing our best to let people know that something like this exists because it is only available across a set number of experiences.

The adoption piece is really about usage. How many people are using it and can we ensure that we are including this feature across more and more surfaces on Pinterest. Over the past year the team has worked hard to surface the feature for a set of queries, you could see these experiences on Pinterest by adding in product education and building out the experience so that it was more apparent in everything that we did.

AT: We had planned a series of marketing campaigns with the broader team internationally. It was the first time that we had rolled out this feature to an international market. It was a big moment for us. Not all the marketing campaigns started at the same time as our public announcement with press and marketing, and we found that every time we ran an in-product marketing campaign we saw bumps in usage of the future. So we continued to do it and figure out other ways to drive awareness as well. What we learned from the experience of this go to market plan was simply that talking about the future brought a lot of awareness publicly because it was so hidden. Like you mentioned, my team launched this feature more broadly in August of 2020, but the feature had been around since 2018 and it was on mobile since 2019. It was hard to find and people weren't aware of it. People didn't know how to get there and if they did they’d forget how to return to it. If you looked at Twitter, you’d see people were asking how they could get back to the feature. What we were trying to do was enable people to know where it was and how to return to it so that they could continue to have these really delightful experiences of finding more content for people like them.

AT: I don't know if there was a thing that I thought that we shouldn't have spent time on. There are definitely things that we realized we needed to spend more time on. Over the last year, there's clearly been a lot of social unrest when it comes to equality in the world and in particular here in the US. With everything that happened, it was really important to the team to do our best to create product experiences that really put Black, Indigenous, and people of color at the forefront of our product decisions.

One of the things that we continue to invest in is making sure that the model that enables us to build the skin tone range feature is really, really good at identifying skin tone in an image. The team is full stack, it's not just a front end user experience. The magic behind all of it is that we have an in-house model that is able to take an image and identify what the predicted skin tone is. It maps that value to a range of colors on the backend that enables us to build a better experience for people because they are able to filter their beauty results on Pinterest by the different skin tone ranges.

Over the past year the learning for us was that by increasing the model’s precision and recall it increased all those other metrics that we were looking at when it came to awareness and usage. The better the model, the more queries we are able to show this product experience on, which leads to more awareness and greater engagement. The team has been spending a lot of time on the model, we continue to iterate and it gets better and better. Those improvements have been really important to the continued success of the team.

The amazing thing about this product is that there are so many cross-functional experiences and cross-functional contributions. From the model improvements to the marketing campaigns we talked about. It’s pretty cool to see everybody contribute in some way, shape or form.

AT: Absolutely. The model that we built is focused on static images and performs really well on static images. Pinterest recently announced the general availability of a new format called Idea Pins, which are really video focused. One of the big challenges from the technology side for the team is to ensure that we can use the model on video pins. It already works, but we can always do better. The opportunity for the team to continue to improve the model is huge and we have a lot that can be done to improve when it comes to video.

I can’t share any specific right now, but we hope to continue to build more experiences that are user facing and that help people, in particular BIPOC folks, feel like Pinterest is a place for them. We hope to be announcing more things later in the year. A lot of our work is going to be user facing, but there will also be a number of improvements we hope to make on the back end, too, when it comes to machine learning and modeling.

AT: Yes, definitely. As I mentioned before, the team originally started off as a virtual team, one person here, one person there and a product manager working on it part time. Now, it's my full time job to work on inclusive products. What that means is that not only do I work with a wonderful team of engineers that works on front end experiences and a team of engineers who works on the ranking and machine learning, but it also means that we, as a team, are setting principals in place to build foundational inclusive product thinking across all of Pinterest. So whether that means, for example, how we think about building new user experiences that are inclusive or how we highlight BIPOC-owned businesses in product, we work with teams across the company to help and share best practices. Inclusive product is built into everything that we do and that has been one of the really interesting things about the past year. I think one of the bright lights for me with the social unrest that has happened in the US in the last year is that it has really enabled the team at Pinterest to figure out how to build inclusive product thinking across everything that we do. It's really important to make sure that my team leads from this place of first principles of what the expectations are of people when coming to Pinterest and ensuring that other teams are empowered to think in the same way.

AT: Absolutely. Since the moment of George Floyd’s death last year, which was absolutely horrific, we have really rallied as a company to ensure we are building experiences that allow people to see themselves in the product. I think the industry has rallied around this idea of figuring out how to make sure that the humans that use our products can feel like they are represented in the platforms that we build. That has been really important. When it comes to the way that we set goals, we have elevated our goals to the top level of the company so that we are held accountable by the entire company for the things that we build. That type of visibility makes it really apparent to everyone that works at Pinterest and I hope also to people outside the walls of our virtual offices that this matters not only to us, but it matters to everyone.

AT: One of the biggest challenges that I have as a PM in this inclusive product space is, how do I prioritize the right set of things? The product space is really new. I’ve spent a decent amount of time talking to PMs at other companies who focus on inclusive products, but every company is slightly different. Pinterest is a visually forward platform where people find inspiration and that visual representation of humans is really important. That's why we focused on building in-house signals like the skin tone signal that enables us to build user experiences that people can touch, feel, understand and control themselves. That kind of control for people of color was really important to us because that says to people "you can control your own experience on Pinterest. We are not going to make assumptions about the person that we might think that you are."

My advice to other people would be simply pick one thing. Take your goals and break them down into bite sized pieces. This is also my advice to the PMs who are new in their careers. Think about how inclusion and diversity applies to your platform and try to take on a small goal and prove that you can do it, because that's what we did. We bootstrapped this feature and found that we could get a lot of momentum because people were excited about it. Now I can sit here and say that we really, as a company, rallied around this idea of inclusive product and it was because we took some small goals and a small spark of idea, proved that it could work and then expanded on what that meant for the broader company. Now my team is not the only team that works on it. There are tons of people who work on this. But I think that you always just have to start small no matter what it is and be really deliberate about why you are picking a specific area. To me, the way that you pick that area is by thinking about what is going to have the most impact for people.

AT: Yes, how you articulate the vision for what you do is very important. We are building experiences for people that are oftentimes underrepresented at tech companies. To be quite frank, I think that in Silicon Valley and in the tech world at large, for product managers everywhere, we are pretty well aware of the fact that Black, indigenous, and people of color are underrepresented in tech. My advice to those leading an effort like this is that the ability to tell the story of your feature is so important because a lot of people cannot feel what it's like to not be represented online. I always open up with the same story when I am telling people why my team exists and why it's important for Pinterest to invest in this as a company. I show a series of fashion magazine covers from the 90s in which every single magazine cover has a skinny white woman. As a woman of color, seeing this in the 90s, it told me that I didn't belong. Then I show what it's like on Pinterest now for me browsing beauty inspiration or fashion inspiration and I see people like me. Telling that story and showing it side by side helps people who may not understand what it's like to not be represented. It is always powerful and it gets people excited about what we can do and what the potential opportunity is to impact not only Pinterest users, but people all over the world.

AT: My LinkedIn is a good place. Also for folks interested in learning about the technical side of how we built our model, check out our Engineering Blog for an overview of the tech, as well as perspectives from the beginning. You can also find more information on our company Newsroom from our announcement last summer and another from earlier this year.

–