Samanyou Garg

GPT Router — Avoid OpenAI downtimes - one API for 30+ LLMs

Featured
60
GPTRouter is an open source LLM API Gateway that offers a universal API for 30+ LLMs, vision, and image models, with smart fallbacks based on uptime and latency, automatic retries, and streaming. Stay operational even when OpenAI is down.

Add a comment

Replies
Best
Samanyou Garg
Hi Product Hunt, Today, we are excited to be open-sourcing GPTRouter, an LLMOps tool we have been using internally at Writesonic for handling millions of monthly requests for our users. ✅ Universal API for 30+ LLMs, Vision and Image Models ✅ Smart Fallbacks based on latency and uptime ✅ Automatic Retries ✅ Supports streaming Since embracing OpenAI GPT-3 in production in 2020, we at Writesonic have been serving millions of users and faced the typical scaling pains with generative AI models: 1. Dependency on a single model risked total downtime. 2. Latency issues with models like GPT-4 affected user experience. 3. Integrating various models was tough due to different APIs and SDKs. 🌟 Early this year at Writesonic, we set out with a clear vision: to become model agnostic. Faced with single-model limitations and diverse AI challenges, we began building GPTRouter - our bespoke solution to navigate and thrive in a multi-model AI world. 🔗 With GPTRouter's Universal API, you're the master of AI models. Swap between OpenAI, Azure, Anthropic, Replicate, Cohere & more with just one line of code. It simplifies model management to a great extent. 🛡️ Downtime isn't an option. GPTRouter's Smart Fallbacks mean your service is always on. You can define a hierarchy of models for each use case. GPTRouter will constantly check for uptime/downtime, latency and other factors, and automatically fallback to the next best model with zero interruption. ♻️ Say goodbye to manual retries. GPTRouter does the heavy lifting with Automatic Retries for failed requests, keeping your AI services sharp and consistent. 🌟 GPTRouter's Edge: ✅ Universal API for seamless model switching. ✅ Smart, automatic fallbacks for continuous service. ✅ Reduced latencies for quick interactions. This is just the starting point. We are also working on integrations with LangChain and LlamaIndex. Additionally, we will also be open sourcing our frontend LLMOps layer that provides a playground to test multiple models in parallel, keep a tab on the latencies for each model, track tokens and costs for each model and user all in one place. 📈 With GPTRouter, we're not just solving our problems at Writesonic; we're offering a solution to the startups and companies looking to make use of Generative AI in production. We are looking forward to seeing how developers leverage GPTRouter in their own use cases. Thank you!
Samanyou Garg
@shivam_tiwari24 Glad you liked it!
Dhairya Aggarwal
The automatic retries and streaming add even more robustness to this already impressive gateway. The fact that it can stay operational even when OpenAI is down is a testament to its resilience and reliability. An absolute must-have tool for anyone working with LLMs. Kudos to the team behind GPTRouter!
Samanyou Garg
@dhairya_ws Thanks, Dhairya!
Mohammad Yousuf
Congratulations on the launch. I was looking for something like this. Just tried to find detailed documentation but can't find it. For example, I want to set 3 Open AI models as fallbacks if my primary model fails. Where should I make the changes?
ANUBHAV GUPTA
Hii @anik_yusuf you can find the example under the heading Managing the Fallback Order here -> https://gpt-router.writesonic.co... please let us know if you encounter anything else
ANUBHAV GUPTA
@anik_yusuf thanks for the complement
Bayazid Malikov
Great to see AI products! Could you tell me about the cost differences?
Samanyou Garg
@bayazid_malikov this open source and totally free.
Matt
@samanyou_garg Very cool concept!
Samanyou Garg
@mattmule Thanks Matt!
Kshitij Mishra
Strengths: Clear value proposition: Highlights the key benefits of avoiding OpenAI downtimes and accessing multiple LLMs with a single API. Technical details: Mentions important features like smart fallbacks, retries, and streaming, appealing to developers. Open source: Showcases transparency and community appeal.
Samanyou Garg
@kshitij_mishra4 Thanks a lot for summarizing it, Kshitij!
Lin Vert
Great work Samanayou! You've made an amazing way to help us stay connected and avoid downtimes with one unique API. Kudos to you for creating this open source powerhouse!
Samanyou Garg
@online_converter thank you, Lin!
Shahmir Faisal
Congrats on the launch Samanyou!
Samanyou Garg
@shahmir_faisal Thanks Shahmir!
SHIVAM SINGH
Wow, this product is a must-have for LLM work! With automatic retries and streaming, it boosts the already impressive gateway. Its ability to stay operational during OpenAI downtimes shows how reliable and robust it is. Hats off @samanyou_garg for this awesome product!
Samanyou Garg
@shivam_singh63 Thank you for your kind words, Shivam!
Jason Todd
Thanks for creating this incredibly innovative tool. I can immediately see several great use cases and I look forward to beginning to play around with it.
Samanyou Garg
@jason_todd123 Do try it out and let us know.
Ethan Stewart
Thank you for the good program
Samanyou Garg
@ethan_stewart Thank you for checking it out!
Siddhant Bansal
This is a great launch. Kudos to the team. <3
Rifah Nawar
This will be really helpful for the AI Tool builders who’re just starting out 🙌
Samanyou Garg
@rifah_nawar4 Yess, thank you!
Aniket Singh
Best in the business
Preethi Anchan
This is a game-changer. Excited to see the difference it brings to developers and AI services!
Samanyou Garg
@preethi_anchan Yes, thank you!
Vikhyat Puri
Woah! This is amazing
Samanyou Garg
@vikhyat_puri2 Yess, thank yoU!
harsh akshit
This can be a game changer in LLM domain. All the best!
Samanyou Garg
@harsh_akshit2 Thank you, Harsh!
Nekane
Awesome Starred on github
Rajesh Kandaswamy
Very cool. This can be quite useful.
Samanyou Garg
@rajeshkan72 Do try and let us know.
Lucas Young
Congrats guys
Samanyou Garg
@lucaxyoung Thanks Lucas!