↑ ↓ Navigate ↡ Select ESC Close

Software Horror Stories | The Standup

2025-07-28 Science & Technology
119.4k
3.0k
211
ThePrimeTime
ThePrimeTime
1.1m subscribers

Unlock all features

FREE: Get instant access to 10 AI summaries, chats, or transcripts per day.

Description

πŸ”— Sponsored by Code Rabbit https://coderabbit.link/primeagen-vscode #sponsored ## πŸ“Œ Chapters: 00:00:00 - Intro & Teejs Medical Records Disaster 00:04:20 - Code Rabbit Sponsor #ad 00:05:04 - Nobody wants to work in Healthcare 00:07:05 - Healthcare testing process 00:12:00 - Epic's Development Pipeline Deep Dive 00:21:33 - Load Testing Nightmare 00:25:00 - The 45-Minute Site Outage 00:27:50 - Class Registration Disaster 00:32:00 - Everything Goes Wrong in Production 00:35:55 - Netflix Static Variable Mistake 00:39:00 - The Boss Who Spotted the Obvious Bug 00:42:00 - Lady Gaga's Broken Countdown Billboard 00:47:00 - 10 Requests /s Disaster 00:49:30 - GraphQL Query Explosion ## Key Topics: - Medical software development challenges and patient safety - Epic Systems development process and MUMPS language - Load testing disasters and production outages - Netflix development war stories - Server-side rendering performance pitfalls - GraphQL resolver optimization nightmares - The importance of proper testing before production releases ## Hosts: - **TJ** (@teej_dv) - Epic Systems horror stories - **Trash Dev** (@trash_dev) - Load testing and Next.js disasters - **Prime** (@ThePrimeagen) - Netflix static variable nightmares

Top Comments (10)

@PaulPaulPaulson 2025-07-28

My horror story: I used a third party DLL and it spawned a thread that repeatedly changed the settings of the floating point unit and this wasn't documented. And no, FPU settings aren't thread local. A week of debugging of seemingly random behavior until I even knew what was happening.

376 9 replies
@bobothefrofro 2025-07-28

My horror story (also in healthcare): I was doing some testing on a data visualization application (in the testing environment, I'm not dumb enough to do this in production), but I needed some more data. My idea: write a MySQL script to randomize the dates of appointments/referrals in the system so they fit within the last 7 days. The script worked perfectly on the first try, checked the data in the database, AWESOME. Refresh the browser in the test environment, no change. Refresh again, no change. Go back to the db tool, and yep, I ran the query on the production connection. Lucky for us, we had the created/updated dates duplicated in a shared/reference table, so I was able to scramble and restore the data before anyone noticed. But yeah, nearly pooped my pants.

127 6 replies
@digitalpacman 2025-07-28

Should have invited me. I worked at an ecom company that had a myspace+facebook app, and, private email servers. An idiot at the company was tasked with "reviewing servers to find unused servers, consolidate resources, and save money by removing servers.". He didn't check the server at all, and sold it back to our hosting provider. The myspace+facebook app, and our private email server was on that box. All email went down, all myspace+facebook went down. Could we get them back? No. No backups were setup. Could we get the server back? No. The hosting company wiped it and sold it already. I hate this guy.

87 6 replies
@Zlei. 2025-08-01

Can't believe Trash Dev took down Google, Instagram, AWS, Sony, & contributed to every other major outage in the past 5 years!

23
@ThePrimeTimeagen 2025-07-31

Hope you enjoy this episode about the craziest horror stories from teej & prime! Thanks to this week's sponsor: Code Rabbit If you're tired of developers just replying with LGTM, then you NEED Code Rabbit. An actually useful application of AI where it can use your own rules, lints and more to provide instant feedback in your editor OR a review on Github. https://coderabbit.link/primeagen-vscode

18 8 replies
@joeylamping7123 2025-08-02

Ex Cerner guy here.. hilarious to hear Epic was going through all the same things we were and a completely proprietary language to boot. Healthcare IT is something else

17 1 replies
@nulifyer4460 2025-08-02

This Epic story is my life. My grandfather worked for the enterprise that made Mumps and I now have his job haha. It was eventually called CachΓ© and is now Inter systems - Iris.

15
@jwr6796 2025-07-28

I'm now just as concerned about my hospital's data infrastructure as I am my bank's. Thanks, TJ!

15
@Chris-se3nc 2025-07-28

Epic’s release pipeline is like an old school Nintendo game. You could potentially lose at a later level and be kicked right back to the beginning. πŸ˜‚

14
@VictorYami 2025-07-29

Code Rabbit is the Raid Shadow Legends of the developer world.

6

Unlock the Data Inside
Turn Videos into Knowledge

  • Get FREE 10/day: transcripts, summaries, chats
  • Chat with videos, export text & PDF
  • $1 free API credit for RAG, chatbots & research

Free forever plan β€’ All features unlocked

App screenshot