The art and science of private alpha testing

Will Little

Assuming you’ve gone through the process of ideating, validating, and creating a new product, now it’s time to get serious about testing it. Hopefully by now engineering has baked in an appropriate amount of automated tests to meet the specifications you put together, but the features you decided on are about to change. Welcome to real life.

When human beings start using software they didn’t code up themselves, the flaws and stupid design decisions become painfully obvious (even if you consider yourself a UX expert).

A fun and scary milestone for a tech startup is when engineering deploys code into a staging environment, solves all the miscellaneous problems that (always) come up, then tells the rest of the team it’s time to check it out.

People have different names for this stage — and the types of testing that should be done here — but this is Private Alpha. It’s meant to give your team a chance to actually try out the functioning product (e.g. on various devices) and ensure all critical bugs are fixed and features are in place that should be there for your business to succeed.

Who exactly should access the private alpha?

After doing this process for more than a dozen years across a variety of tech platforms and business models, I’d highly recommend including your advisors and most-friendly customers in the private alpha group. By “most-friendly” I mean people who understand this will be a clunky experience, empathize with the situation, and won’t freak out that you are behind schedule and the product doesn’t work very well.

The reason why you need these people clicking and swiping around at this point is because their feedback is critical. Hearing (and inviting) the brutal truth from the people who didn’t design or engineer the product is exactly what needs to happen at this point. You don’t want to skip this step and then give your first beta testers a terrible first impression (it will still be terrible, of course, but you can make it less terrible with a proper private alpha stage). 

What to do with the feedback

Sometimes you can get lucky and your alpha testers are impressed and have very little feedback, but that is rare. In fact, if you don’t hear the negative impressions then you need to ask them to dive deeper and shoot straight with you. Once you’ve assembled all the feedback into specific tasks/stories in your project management tool, now you need to do a massive gut check: 

• Are alpha testers going to be excited about sharing this with their friends and colleagues? (Yes, ask them directly and calculate your rough Net Promoter Score) 
• Can your alpha testers give you the names and contact info of other people who could be customers? (If they are hesitant, this is a warning sign and you need to dig into why)
• Tell them your product roadmap and timeline as a conversation starter for their feedback. Are they enthusiastic about your direction? If not, this is also a warning sign. 

Widen the private alpha circle in 3–4 iteration

You see what we’re doing here, right? We’re using the private alpha stage as yet another mechanism to acquire customers. Ask your advisors, team members, and early customers for recommendations of good friends who would want to try this out and give you brutally honest feedback. Repeat this process three or four times until you have a decent number of empathetic testers poking around and running into bugs on their various devices. 

Fix these bugs, record their feedback, and take the time necessary to put more polish on your product. Assuming you are moving forward in agile fashion, hopefully you haven’t set a hard-and-fast launch date yet. Many startups succeed or fail precisely at this fragile stage, usually by not getting enough of the brutally honest feedback from their alpha testers that secretly are thinking “meh” (which equals death for a startup). 

Author’s note, in the next couple articles in this series I’ll address the operational nuances and strategy regarding the landing page, private beta, launch day, and public beta stages. Subscribe to my newsletter to stay posted. Also, thousands of startup founders leverage a web application that my partners and I built on top of the “ideate, validate, create, grow, fund” operations framework that this article series describes. We put the framework together from decades of experience at both sides of the funding table. We also have a private Slack community, custom mentoring services, and a ton of resources for early stage founders. Check it out.