Case Study
Leading Successful Cost Optimization at CMS
Streamlining management, building sustainability, and achieving cost efficiency for CMS
The client
Centers for Medicare and Medicaid Services
Cost Optimization Assessment
Challenge
For over five years, Flexion has led cost optimization efforts at the Center for Medicare and Medicaid Services (CMS), integrating cost-conscious strategies to the forefront of architecture, engineering, and strategy-based decisions. Recently our team was tasked with reviewing the AWS infrastructure, pinpointing cost-saving opportunities, and providing recommendations to Application Development Organizations (ADOs). However, communicating potential overspending to ADOs, especially in the presence of government officials, is a delicate task. The contracting world can be sensitive to discussions around inefficiencies, making it challenging to convey our findings without creating friction.
Our approach
Image 1: Cost Assessment Format
To prioritize cost optimization in CMS’s engineering and leadership, we adopted a comprehensive approach: “Research -> Review -> Alert/Monitor -> Culture Change.” (image 1) This multi-step strategy ensures cost awareness and sustainable practices are top-of-mind for engineering teams.
Image 1: Cost Assessment Format
Research
We began our consultative overview by thoroughly researching the overall cost of the AWS account. This included analyzing the overall spending, including Reserved Instances and Savings Plans (RI/SP) coverage. We developed scripts to gather cost data efficiently while also deep-diving into every aspect of the account. We compiled valuable Cost Data for review with the account while also providing expert-level recommendations for improvements.
Machine and Database Utilization
We identified excess costs in the use of Elastic Compute Cloud (EC2) and Relational Database (RDS). We analyzed EC2 Central Processing Unit (CPU) usage to ensure proper instance sizing and often found overprovisioned instances. Using a Cloudwatch API script, we gathered average and maximum CPU statistics to spot underused resources. For RDS, we reviewed CPU usage and active connections to confirm their necessity. We flagged instances without active connections for potential termination to cut costs.
S3 Usage
AWS S3 Storage Lens helped us review S3 usage and cut storage costs. We emphasized lifecycle and version policies for cost control. We also reduced expenses by deleting outdated versions of data files. As a part of our review, we also generated a list of S3 buckets with their prospective lifecycle and version policies via a script.
Spending Deep Dive
We reviewed the top 10 AWS expenses and billing in depth. We identified and resolved issues like indefinite log retention in Cloudwatch where we had to work with CMS officials to establish the correct retention policy with cost in mind.. We also analyzed Trusted Advisor findings to implement targeted cost-saving actions.
Recommendations
We categorized our recommendations into quick fixes and long-term changes, ensuring a clear path forward for cost optimization. For each recommendation, we calculated potential monthly and yearly savings, offering CMS a concrete view of the financial impact. To streamline implementation, we provided SME guidance and, for example, JIRA tickets to support quick ADO adoption into their program increments. In discussions with CMS leadership, we prioritized the highest-impact savings, focusing on results that would make the biggest difference.
Review
We gathered our findings and presented them to ADOs, addressing data discrepancies to ensure everyone had a clear understanding of the next steps. With CMS leadership’s backing, we set specific cost optimization goals that aligned with the project’s vision. To keep momentum, we scheduled regular check-ins, allowing us to support ongoing implementation and adjust strategies as needed. JIRA tickets became our roadmap, tracking each step of progress and keeping our goals in sight.
Alert/Monitor
To further support the cost-conscious culture, we set up continuous alerts to monitor idle EC2 usage, Trusted Advisor warnings, and budget anomalies, ensuring nothing slipped through the cracks. Each day, the automation we created sent out email alerts on idle resources, prompting swift action. This consistent monitoring sharpened the team’s responsiveness, turning insights into immediate action. With these alerts in place, we enabled proactive cost management across all CMS accounts, keeping costs controlled and the system efficient.
Image 2: Idle Alert Architecture
We collaborated with ADO management and CMS Infrastructure Contractors to activate weekly Trusted Advisor and Budget Alerts within the AWS Console, designed to monitor cost optimization and spending anomalies. By setting these alerts to 10% above the average monthly spend and labeling them as “Anomaly” alerts, we could proactively catch irregular expenses before they impact the budget, ensuring efficient cost management with timely notifications.
Culture Change
We cultivated a cost-conscious culture that emphasized growth and improvement rather than blame. By using positive language like “Potential Savings,” we created a supportive environment where teams felt safe to address and reduce costs openly. This shift empowered teams to take on cost-saving efforts with the support of CMS, transforming cost management into a shared responsibility. Over time, our approach allowed cost management to thrive independently, with teams driving sustainable practices that required minimal oversight.
Image 3: The Safe Terminology
Technology stack
Outcomes
We embraced cost optimization’s heightened importance in government requirements by openly addressing pushback, welcoming diverse perspectives, and positioning ourselves as thought leaders in AWS and engineering best practices through transparent collaboration.
In the first four months of 2022, we completed the “Cost Assessment Process” with two Application Development Organizations (ADOs) and initiated it with two more, achieving an annual savings of $538,624. CMS/ISG leadership responded with strong support, enabling us to expand this initiative to over 20 additional ADOs. By fostering a collaborative, psychologically safe environment, we built lasting relationships with teams, establishing ourselves as trusted partners across CMS/QNET and other CMS subgroups. Our ongoing meetings with various CMS organizations allow us to share valuable insights and showcase the effectiveness of our cost-saving strategies. This work has helped CMS lower management overhead, reduce environmental impact, and promote sustainability throughout its operations.
Ready to change the way you’re doing business?
Contact us to discuss how Flexion can help your organization become more cost-efficient and productive.
A proud AWS partner.
