Our client, a dynamic Holding Company based in Orlando, Florida, USA, specializes in venture capital and private equity. With a diverse portfolio spanning lead generation, market analysis, financial services, and innovative technology solutions like targeted email delivery, they're at the forefront of driving success across multiple industries. Since its establishment, the institution has strategically utilized AWS but encountered optimization challenges, which is where NeenOpal stepped in to optimize their AWS workloads.
The removal of console access to the AWS Data Pipeline service from May 12, 2023, presented a significant challenge for the client. This change required them to take proactive steps to migrate and adapt to alternative solutions within a specified timeframe.
The substantial cost associated with Glue Jobs linked to DynamoDB amounting to $5,000 per month, presented a financial hurdle for the client. Finding a cost-effective alternative or optimizing the existing setup became crucial to alleviate the economic burden and enhance operational efficiency.
The client uses AtData for Email Intelligence and the slow data request and retrieval often leads to loss of opportunities as contacts become inactive after a day, hampering their ability to capitalize on potential leads and maximize engagement.
In this case study, NeenOpal leverages its expertise in AWS services to provide a solution tailored to excellence. By optimizing Glue jobs, utilizing Lambda, and maximizing efficiency with Redshift, we guarantee seamless data processing, deliver significant cost savings and heightened efficiency for the client’s operations. Key initiatives involve migrating 150 ETL pipelines to AWS Glue, refining Glue jobs to minimize redundant executions, and addressing data processing delays.
Migrating approximately 150 ETL pipelines, which encompassed multiple data sources and involved four different databases, to AWS Glue was a pivotal step. Our template-based approach expedited and enhanced the workflow. The entire process, including script creation, job configuration, and workflow establishment from scratch, was executed based on existing rules. A meticulously planned switch-over process – encompassing stopping, starting, and backfilling – ensured a seamless transition. The databases involved in this migration spanned both RDS and RedShift.
Recognizing that the data schema had remained unchanged in the last 3-4 months, we optimized the AWS Glue jobs by reducing unnecessary Crawler execution. This was achieved by modifying the cadence from daily to monthly and triggering execution only when required, such as in the event of a process failure. Additionally, to enhance efficiency, Glue jobs were divided into two distinct parts: DynamoDB to S3 and S3 to RedShift upload. The former utilized data dump features in DynamoDB, while the latter leveraged the Copy command within RedShift. This optimization resulted in a remarkable reduction of unnecessary DPUs from 10 to 0.0625 which in turn resulted in a major cost reduction.
To address the lag in sending contact details to AtData and receiving traffic information due to prolonged processes (12 hours of sending and 12 hours of processing), we implemented a strategic solution. The process was dissected into smaller, more manageable pieces – processing 50 chunks of data per hour. This not only facilitated the receipt of traffic/activity details within an acceptable time range when the subscriber is online but also incorporated a feature to decline information from the AtData when the contact was no longer active. These improvements were seamlessly integrated using AWS Lambda functions, contributing to enhanced efficiency and real-time data processing.
NeenOpal Inc. is an AWS Advanced Services Path, Differentiated Partner and we are in the Public Sector and Well Architected Partner programs. As a forward- thinking consultancy, we specialize in unlocking the transformative power of data to drive business growth. Our team comprises AWS Certified experts committed to staying abreast of the latest industry advancements.
All 150 ETL pipelines, vital for the live business application, were smoothly migrated to AWS Glue within just 1.5 months, with no visible downtime for customers. This seamless transition not only showcases the effectiveness of our migration strategy but also guarantees uninterrupted business operations.
The challenge posed by the impending decommissioning of AWS Data Pipeline service was effectively mitigated. With all pipelines now operating through AWS Glue, there is no longer any dependency on the discontinued service. This strategic shift has enhanced the reliability and sustainability of the data infrastructure.
Significant financial relief was attained by optimizing DynamoDB-related Glue costs, reducing monthly expenses from around $5,000 to a sustainable $400-$500. This represents a substantial cost saving measure for the customer, highlighting the success of financial efficiency initiatives.
The client's mailing system underwent optimization, including status checks at each stage, reducing delays and enhancing reliability. Batch processes replaced real-time retrieval, speeding up contact targeting and cutting costs by blocking extraneous data, ultimately streamlining operations for improved cost-effectiveness.
In conclusion, the successful execution of this comprehensive solution underscores the transformative power of advanced analytics and robust data processing methodologies. By optimizing budgets and enhancing overall performance, the initiative not only achieves immediate objectives but also lays a solid foundation for future growth and innovation.
Written by:
Senior Associate Consultant
Content Writer