Unlocking Efficiency: How We Transformed Workloads and Slashed Costs for a Dynamic Holding Company

AWS Data Pipeline Transition

The removal of console access to the AWS Data Pipeline service from May 12, 2023, presented a significant challenge for the client. This change required them to take proactive steps to migrate and adapt to alternative solutions within a specified timeframe.

Optimizing DynamoDB-Linked Glue Jobs

The substantial cost associated with Glue Jobs linked to DynamoDB amounting to $5,000 per month, presented a financial hurdle for the client. Finding a cost-effective alternative or optimizing the existing setup became crucial to alleviate the economic burden and enhance operational efficiency.

Overcoming Data Retrieval Delays

The client uses AtData for Email Intelligence and the slow data request and retrieval often leads to loss of opportunities as contacts become inactive after a day, hampering their ability to capitalize on potential leads and maximize engagement.

01.

ETL Pipeline Migration to AWS Glue

Migrating approximately 150 ETL pipelines, which encompassed multiple data sources and involved four different databases, to AWS Glue was a pivotal step. Our template-based approach expedited and enhanced the workflow. The entire process, including script creation, job configuration, and workflow establishment from scratch, was executed based on existing rules. A meticulously planned switch-over process – encompassing stopping, starting, and backfilling – ensured a seamless transition. The databases involved in this migration spanned both RDS and RedShift.

02.

Optimization of AWS Glue Jobs

Recognizing that the data schema had remained unchanged in the last 3-4 months, we optimized the AWS Glue jobs by reducing unnecessary Crawler execution. This was achieved by modifying the cadence from daily to monthly and triggering execution only when required, such as in the event of a process failure. Additionally, to enhance efficiency, Glue jobs were divided into two distinct parts: DynamoDB to S3 and S3 to RedShift upload. The former utilized data dump features in DynamoDB, while the latter leveraged the Copy command within RedShift. This optimization resulted in a remarkable reduction of unnecessary DPUs from 10 to 0.0625 which in turn resulted in a major cost reduction.

03.

Mitigating Lag in Data Processing

To address the lag in sending contact details to AtData and receiving traffic information due to prolonged processes (12 hours of sending and 12 hours of processing), we implemented a strategic solution. The process was dissected into smaller, more manageable pieces – processing 50 chunks of data per hour. This not only facilitated the receipt of traffic/activity details within an acceptable time range when the subscriber is online but also incorporated a feature to decline information from the AtData when the contact was no longer active. These improvements were seamlessly integrated using AWS Lambda functions, contributing to enhanced efficiency and real-time data processing.

Seamless Migration & Zero Downtime

All 150 ETL pipelines, vital for the live business application, were smoothly migrated to AWS Glue within just 1.5 months, with no visible downtime for customers. This seamless transition not only showcases the effectiveness of our migration strategy but also guarantees uninterrupted business operations.

Mitigation of Data Pipeline Inaccessibility

The challenge posed by the impending decommissioning of AWS Data Pipeline service was effectively mitigated. With all pipelines now operating through AWS Glue, there is no longer any dependency on the discontinued service. This strategic shift has enhanced the reliability and sustainability of the data infrastructure.

Substantial Cost Reduction in DynamoDB-related Glue Costs

Significant financial relief was attained by optimizing DynamoDB-related Glue costs, reducing monthly expenses from around $5,000 to a sustainable $400-$500. This represents a substantial cost saving measure for the customer, highlighting the success of financial efficiency initiatives.

Streamlining the Email Delivery Application Process for Enhanced Efficiency

The client's mailing system underwent optimization, including status checks at each stage, reducing delays and enhancing reliability. Batch processes replaced real-time retrieval, speeding up contact targeting and cutting costs by blocking extraneous data, ultimately streamlining operations for improved cost-effectiveness.