Hey Everybody 👋 , We are excited to open source an amazing tool that we've been using internally for some time and which helped us a lot in most of our AI projects - Deepmark AI! Deepmark AI is a benchmarking tool for GenAI builders that enables assessment of several large language models (LLM) on various extrinsic (task-specific) metrics (e.g. accuracy, relevance, failure rate, latency, etc) on your own data, so your AI applications have predictable and reliable performance. 🎯 Why we building this ? Based on IngestAI Labs' 40'000 users and dozens of AI projects we completed, we clearly see that GenAI builders and organizations need to be able to assess large languag models (LLM) on their own data to deliver predictable and reliable results that balance accuracy, precision, recall (the model’s ability to correctly identify positive cases within a given dataset), as LLM models can produce different answers to the same prompts, impeding the user’s ability to assess the accuracy of outputs. To address this challenge of reliability, we (IngestAI Labs) have developed and open sourced Deepmark AI - a benchmarking tool, that enables assessment of several large language models (LLM) on various extrinsic (task-specific) metrics on your own data. It has pre-built integration with leading Generative AI APIs such as GPT-4, Anthropic, GPT-3.5 Turbo, Cohere, AI21, and others. 🚀 DeepMark AI is a tool specifically designed for AI builders.This solution focuses on iterative assessment of extrinsic metrics to identify predictable, reliable, and cost-effective Generative AI models based on the unique needs of a particular use case and data. Deepmark AI offers comprehensive assessment of various important performance metrics such as: ⚡Question answering accuracy ⚡Text classification accuracy ⚡PII recognition accuracy ⚡Named entity recognition (NER) accuracy ⚡Summarisation quality (Relevance) ⚡Sentiment analysis accuracy ⚡Cost analysis ⚡Failure rate ⚡Accuracy ⚡Latency Deepmark AI empowers organizations to make informed decisions when navigating through the most important performance metrics of Large Language Models. ❤️ Our Mission : Our mission is to make Generative AI more accessible, organised, and useful for anyone and for any size of organisations. 🎯 Follow our roadmap on Github : Star us / watch us / fork us: https://github.com/IngestAI/deep... And join our community at : 📣 Discord : https://discord.gg/kMpbueJMtQ 📣 Twitter : https://twitter.com/ingestai_io Would love to hear your usecases & feedbacks. 🙌 Hope you loved it. ❤️ Team IngestAI Labs / Deepmark AI

Vasyl R.

Deepmark AI

Maker

📌