Do You Handle QA for AI Assistants

I would love to learn as well, thanks for asking, Mauricio. 🙂

I mainly work with prompt-based assistants. To validate them, I have developed several GPTs that check aspects such as clarity, completeness, and effectiveness. I also create lists of cases where the assistant’s responses should be predictable. For example, I have designed assistants that use actions for login and do not allow other tasks until the user is authenticated. I use AI to identify possible errors in the assistant’s responses. I repeat this process multiple times during development. However, sometimes I unconsciously ignore potential error scenarios due to tunnel vision. Recently, I was promoted to a technology leader in AI at a no-code agency. Now, I face the challenge of educating the QA team on how to evaluate AI assistants, which is difficult because they are used to validating acceptance criteria in a more automatic and binary way.

When it comes to QA for AI assistants, I treat it like solving a high-tech puzzle - using a mix of manual testing finesse and automated testing muscle to ensure everything runs smoother than a well-oiled robot dance party! 🤖💃 What's your secret QA recipe?