Leading successful cost optimization at CMS

Challenge

For over five years, Flexion has led cost optimization efforts at the Centers for Medicare & Medicaid Services (CMS), integrating cost-conscious strategies at the forefront of architecture, engineering, & strategy-based decisions. Recently our team was tasked with reviewing the AWS infrastructure, pinpointing cost-saving opportunities, & providing recommendations to Application Development Organizations (ADOs). However, communicating potential overspending to ADOs, especially in the presence of government officials, is a delicate task. The contracting world can be sensitive to discussions around inefficiencies, making it challenging to convey our findings without creating friction.

Approach

To prioritize cost optimization in CMS’s engineering & leadership, we adopted a comprehensive approach.

Research
We began our consultative overview by thoroughly researching the overall cost of the AWS account. This included analyzing overall spending, including Reserved Instances & Savings Plans (RI/SP) coverage. We developed scripts to gather cost data efficiently while also deep-diving into every aspect of the account. We compiled valuable cost data for review with the account while also providing expert-level recommendations for improvements.

• Machine & database utilization: We identified excess costs in EC2 & RDS. We analyzed EC2 CPU usage to ensure proper instance sizing & often found overprovisioned instances. Using a CloudWatch API script, we gathered average & maximum CPU statistics to spot underused resources. For RDS, we reviewed CPU usage & active connections to confirm necessity. We flagged instances without active connections for potential termination to cut costs.
• S3 usage: AWS S3 Storage Lens helped us review S3 usage & cut storage costs. We emphasized lifecycle & version policies for cost control. We also reduced expenses by deleting outdated versions of data files. As part of our review, we generated a list of S3 buckets with their prospective lifecycle & version policies via a script.
• Spending deep dive: We reviewed the top 10 AWS expenses & billing in depth. We identified & resolved issues like indefinite log retention in CloudWatch—working with CMS officials to establish the correct retention policy with cost in mind. We also analyzed Trusted Advisor findings to implement targeted cost-saving actions.
• Recommendations: We categorized recommendations into quick fixes & long-term changes, ensuring a clear path forward. For each recommendation, we calculated potential monthly & yearly savings, offering CMS a concrete view of financial impact. To streamline implementation, we provided SME guidance and created JIRA tickets to support quick ADO adoption into their program increments. In discussions with CMS leadership, we prioritized the highest-impact savings.

Review
We gathered our findings & presented them to ADOs, addressing data discrepancies to ensure a clear understanding of next steps. With CMS leadership’s backing, we set specific cost optimization goals aligned with the project’s vision. To keep momentum, we scheduled regular check-ins, allowing us to support implementation & adjust strategies as needed. JIRA tickets became our roadmap, tracking progress & keeping goals in sight.

Alert/Monitor
We set up continuous alerts to monitor idle EC2 usage, Trusted Advisor warnings, & budget anomalies. Daily automation sends email alerts on idle resources, prompting swift action. With these alerts, we enabled proactive cost management across CMS accounts.

We collaborated with ADO management & CMS Infrastructure Contractors to activate weekly Trusted Advisor & Budget Alerts within the AWS Console, designed to monitor cost optimization & spending anomalies. By setting alerts to 10% above average monthly spend & labeling them as “Anomaly” alerts, we proactively catch irregular expenses before they impact the budget.

Culture change
We cultivated a cost-conscious culture that emphasized growth & improvement rather than blame. By using positive language like “Potential Savings,” teams felt safe to address & reduce costs openly—transforming cost management into a shared responsibility that now thrives with minimal oversight.

Outcomes

We embraced cost optimization’s heightened importance in government requirements by openly addressing pushback, welcoming diverse perspectives, & positioning ourselves as thought leaders in AWS & engineering best practices through transparent collaboration.

In the first four months of 2022, we completed the Cost Assessment Process with two ADOs and initiated it with two more—achieving annual savings of $538,624. CMS/ISG leadership responded with strong support, enabling expansion to 20+ additional ADOs. This work helped CMS lower management overhead, reduce environmental impact, & promote sustainability across operations.