"Their engineers are phenomenal"
Dana Incorporated is a Fortune 500 company that supplies integrated drivetrain and propulsion systems. In 2020, the company approached us with a challenge: they wanted to improve their strategic decision-making processes by obtaining detailed reports that analyze millions of different data pieces. Our team rose to the challenge and created a custom web scraping solution that helped Dana achieve their goals.
Discovered average competitor prices for specific products.
Identified new features in competitors' similar products.
Gained insight into global electric car demand trends.
Automated pipeline for up-to-date information and reports.
Our solution involved building 25 custom web scrapers for competitor websites and gathering around 12 million unique product data pieces.
We also created an ETL script that filled Dana's internal databases with standardized, aggregated, and loaded data into cloud databases. The gathered data helped Dana gain competitive insights into the automotive market and make informed decisions.
Dana's project presented several challenges that we had to overcome. Firstly, we had to scrape competitor websites that had highly complex forms with around 1000 different permutations of options chosen.
Secondly, to prevent access restrictions, we rotated our IP and used proxies while limiting the speed of each scraper to follow the rules of robots.txt.
Finally, we had to run ten machines asynchronously and on each machine multiple scrapers, which resulted in ~250 crawlers running simultaneously.
In conclusion, our data scraping and monitoring solution proved to be a game-changer for Dana, providing them with invaluable insights and enabling them to make informed decisions to stay ahead of their competitors in the ever-evolving automotive market.
By leveraging our custom web scrapers and advanced data analysis techniques, we helped Dana analyse millions of pieces of product data from their competitors' websites, providing them with pricing, product, market, and brand intelligence.
Our solution not only saved time and resources but also improved the accuracy and completeness of the data, leading to better decision-making and strategic planning.
If you're ready to boost your business, we'd be thrilled to work with you. How to start?