Technology is an interconnected global ecosystem. When a major provider experiences a problem, the ripple effects can be felt everywhere. Recent outages within the last week at both Amazon Web Services (AWS) and Microsoft Azure have had precisely this cascading effect on technology globally. Industries such as travel, e-commerce, finance, healthcare, and field services have all been impacted—and that includes TEAM Software solutions.
We know these disruptions are more than an inconvenience; they are critical interruptions to your business. The Microsoft Azure outage alone impacted the delivery of our DailyPay, Lighthouse, and Hire synchronizations to WinTeam. It also affected the API connection between Symmetry and WinTeam’s automated payroll tax engine, impacting payroll processing on October 29th.
We believe in transparency. In this article, we’d like to share the role these foundational systems play in delivering our solutions, how we communicate incidents to you, and how we’re using events like this to learn, mitigate, and ultimately reduce the impact on your operations.
The Role of AWS and Azure in Delivering Our Solutions
You may wonder why we, and so many other software companies, build on platforms like AWS and Microsoft Azure. These "hosting providers" are a cornerstone of modern software development for several key reasons:
Infrastructure Management: They manage the complex underlying infrastructure—the physical servers, networks, and data centers. This allows our developers to focus on what matters most: building and improving the TEAM Software applications you use every day.
Cost-Effectiveness: We don't have to be in the business of maintaining costly physical servers, allowing us to invest those resources back into product innovation.
Scalability & Performance: Hosted platforms allow us to easily scale services up or down. This is crucial for handling changes in demand and "traffic spikes," ensuring the system remains responsive when you need it most.
Advanced Security: These providers offer robust, advanced security measures that are far more sophisticated than what most individual companies could build, better protecting our clients from advanced attacks and bad actors.
Using these services allows us to be more agile and resilient, but it also means we are part of a larger technology supply chain.
Communicating Outages and Interruptions
We understand how critical our solutions are to your daily operations. When a disruption occurs that impacts multiple clients, it becomes our absolute priority, and our teams work tirelessly to ensure systems are restored quickly and efficiently.
Equally important is your need to know what's happening. Here is our general communication process during an incident:
Alert & Escalation: Once we are alerted to an issue (either through internal monitoring or a client report), we immediately escalate it to our engineers.
Identification & Initial Notice: Our engineers work to identify the root cause. We then send an initial message to our clients, letting you know we've detected the issue and are actively working on a fix.
Periodic Updates: For events that last longer than normal, we will provide periodic updates throughout the outage, keeping you abreast of what we know and our progress.
Resolution: Once we’ve resolved the issue and verified the environment is stable, we send a final communication letting you know the issue has been corrected and the system is restored, allowing you to resume normal operations.
How to Get the Latest Information
We want to make sure you get these communications. If you feel you're missing our notifications, here are two key things you can do:
Check Your Email Preferences: If you’re not receiving our communications regarding service disruptions, it may be due to your email settings. We advise that you take a minute to review your email preferences and select the solutions for which you’d like to receive product updates and technical notices.
Bookmark the Health Dashboard: A great place to start when you first detect an issue is the TEAM Software Health Dashboard. This site provides a real-time, product-by-product status of our solutions, as well as updates on active incidents, active maintenance, and scheduled maintenance.
Future-Proofing: Learning from Every Event
Our work isn't done when the service is restored. Our Engineering teams treat every outage event as a critical learning opportunity.
After each event, our team engages in a comprehensive Root Cause Analysis (RCA) process. In this review, we ask the hard questions regarding our people, process, and technology to understand the "why" behind the event. From this analysis, we create actionable steps to improve our architecture, monitoring, and response plans moving forward.
Technology disruptions are challenging, but our commitment is to navigate them with transparency and a focus on continuous improvement. We value your partnership and are dedicated to strengthening our systems and processes to better serve you.