In response to the review bug
Hello Instructors,
We hear you, and we deeply understand your frustration. We know this bug has impacted your courses and businesses, and we are committed to resolving it as quickly and effectively as possible.
I want to provide a more detailed update based on the latest information from our engineering team. After a thorough investigation, we’ve identified that the issue was caused by a bug introduced into the data pipelines on September 20th. This bug affected how ratings were displayed on course landing pages and impacted search positioning and course discovery.
Here’s where we stand right now:
- A fix has been deployed to correct the rating displays on your course landing pages. Some of you may have already noticed improvements.
- Our team is working on a fix for search positioning and discovery, which is crucial for your course visibility. This will take until tomorrow to fully execute, and our engineers will monitor closely to ensure the quality of this fix.
We’ve also heard your feedback loud and clear regarding communication. Going forward, we’ll be providing daily updates on progress, even if there’s no significant change. We understand that the lack of communication has contributed to feelings of frustration, and we are committed to improving transparency.
Next steps:
- We’ll continue to monitor the situation after the fix is deployed and will keep you informed every step of the way.
- If you are still seeing issues with your ratings after tomorrow, please reach out to our support team directly so we can escalate those cases for review.
Thank you for your patience and continued support. We know how important this is to you and your business, and we are doing everything we can to resolve the issue quickly and fairly.
Thank you,
Udemy Community Team
Comments
-
Thanks @MarinaT
1 -
Thank you, this is very helpful ♥️
1 -
Thank you, @MarinaT, this is great news. The dev team working on a fix for the positioning is excellent, as I'm sure a lot of the instructors were nervous about losing their top places due to the fluctuations (@ScottDuffy the titans like you remain safe!! 😂).
While the incident made a lot of people age a few years in these past few days, it's good to see things back to normal. Thank you for your updates and establishing that bridge with the internal teams!
0 -
@MarinaT thank you for the communication.
My only question is: Why did this change go to production on a Friday (September 20th).
It's a very simple and common engineering practice to avoid pushing change before a weekend or a holiday.
If the engineers at Udemy followed this best practice it would have shortened the duration of this issue by 2 days, which means our business would have suffered 2 fewer days of losses.
8 -
@MarinaT Thank you for the information, this explains why even my course rating went up, I don't have any enrollments (my previous discussion), so the bug is not only the rating but also the course visibility, I hope everything can go back to normal or even better soon. Thank you again.
0 -
Thanks very much for the efforts and taking prompt action! Really appreciate
0 -
Thank you @MarinaT !
0 -
@MarinaT , could you give some color to what happened from a process perspective? How was the issue categorized initially? From an outside perspective, it looked like the issue was detected on Friday, but only started to be dealt with on Monday. Why wasn't this issue categorized as urgent and tackled by the on-call teams during the weekend?
Like most issues, the instructors saw it first (we are your best QA testers and monitors). However, I feel we lack a better channel to share such situations. Is this something that could be worked on?
1 -
Thanks @MarinaT, looks like everything is restored to normal now.
0 -
It does seem like everything is resolved now. Thanks to the Support teams.
It would help to know what exactly the bug was, and how the team is going to ensure that the next time such a thing happens, we don't lose 5-6 days before things get resolved.
1 -
I think that is excellent, thank you @MarinaT.
What would be even better is some sort of root cause analysis and process lookback.
I know that would be good for Udemy, and it would likely make instructors less anxious.- How did it happen?
- What can we do to avoid similar incidents again?
- Process review is critical not just to make sure this mistake does not happen again, but to ensure future one does not because the processes were made better.
- Which processes should be updated/tweaked?
- No updates before weekends.
- Solid QA before rollout in test.
- Rollouts in small batches to observer inadvertent changes.
- Rollback plans.
- Communication protocols both internally, to instructors, and externally.
- etc.
Basically proper change management and project management. Please and thank you :)
5
