Navigate Select ESC Close

My biggest failure to date

2025-06-05 Science & Technology
90.8k
2.7k
253
Theo - t3․gg
Theo - t3․gg
539.0k subscribers

Unlock all features

FREE: Get instant access to 10 AI summaries, chats, or transcripts per day.

Description

We just had our first major outage with T3 Chat. I want to be super clear: T3 Chat downtime is not acceptable. We're doing everything we can to make sure this does not happen again, and I want to be super transparent about exactly what happened. Thank you WorkOS for sponsoring! Check them out at: https://soydev.link/workos Use code BLAME-THEO for 1 month of T3 Chat for just $1: https://soydev.link/chat SOURCE https://x.com/theo/status/1929264138544615464 Want to sponsor a video? Learn more here: https://soydev.link/sponsor-me Check out my Twitch, Twitter, Discord more at https://t3.gg S/O Ph4se0n3 for the awesome edit 🙏

Top Comments (10)

@twilightdev 2025-06-05

Theo will now create FailureThing to prevent such failures

836 5 replies
@cleasbycode7506 2025-06-05

Failure is a blessing in disguise. Look at all the issues it has brought to light, which are now going to get fixed, vastly improving all the related services.

87
@gaganref 2025-06-05

I love these longest deep dive videos more than the shorter videos, almost no one does this except some tech talks and in events. Thanks for taking time and putting these out.

229 5 replies
@JoeMilbachOnline 2025-06-08

I love that you are so transparent. You are the best, keep on keeping on.

8
@JoshuaMelville 2025-06-05

Convex provides cool functionality, but issues like these remind me why it’s often better to stick with known quantities and progressively enhance for things like realtime. Very few apps are mostly realtime.

15
@hazington-dev 2025-06-05

Your next client JS code should listen to two parameters in the response header: app-version and force-reload. That's the way how I ensure that all clients use the most recent version. Without force-reload a non-removable banner appears and let's the user know that the app was updated and must be reloaded. When force-reload is set, then the client will be reloaded immediately, when there's no pending form or when there was no activity for more than 15 minutes. When using a service worker, it's a good idea to add a cache ID calculated from the app-version to the JS and CSS resource URLs. Another benefit of the service worker: When force-reload is triggered in one tab or the user clicks on the update banner, then you can reload all other open tabs. I implemented this mechanism a couple of years ago, when I had my first issues caused by outdated client JS code.

15
@steve-adams 2025-06-05

It's fascinating to see how this was built. I've been building something similar for work (an internal tool, so facing less scale and smaller workloads) and I took an entirely different route. We're both doing the right thing for what we need, so I'm not passive aggressively criticizing. It's just cool to see how different solutions can be when the end goals are so similar.

5
@AnthonyRiceMusic 2025-06-06

Thank you for showing us an actual case study of how things can go wrong in production. Thanks for making this video so can all learn. ❤❤

0
@DiegoDTD 2025-06-08

Thank you Theo, learning a lot, discovering unknowns unknowns. Will apply this on my work :)

0
@wstein389 2025-06-05

This channel is so unique and amazing. Thanks!!!🎉

1

Unlock the Data Inside
Turn Videos into Knowledge

  • Get FREE 10/day: transcripts, summaries, chats
  • Chat with videos, export text & PDF
  • $1 free API credit for RAG, chatbots & research

Free forever plan • All features unlocked

App screenshot