AutoArena

Hey Product Hunt! 👋 I'm thrilled to introduce AutoArena! It's an open-source tool that takes the hassle out of comparing generative AI systems, making the process faster, cheaper, and more accurate. AutoArena uses LLM judges to automate and scale head-to-head evaluations, allowing you to quickly generate leaderboards for different GenAI models, RAG setups, or even specific prompt tweaks. Whether evaluating multiple models or experimenting with new prompt variations, AutoArena can help you clearly understand which setups perform best—without manual scoring. Plus, you can fine-tune your own LLM judges to tailor evaluations to your unique requirements. I’m excited to see how this will help the community push the boundaries of generative AI! And congrats to the team: @gordonhart, @moelgendy, @andrewshishi, and @skip_everling. Give it a try, and I’m sure the team would love any feedback, thoughts, or ideas for new features! You can download it from the Github repo and install it locally, or sign up for the hosted version at https://www.autoarena.app. Check it out and let's automate those evals! 🤖🎉