InfoArmor Tracks 2 Billion Dark Web Credentials with MariaDB

About InfoArmor

InfoArmor (part of Allstate) is a premier provider of identity protection and advanced threat intelligence services. To proactively safeguard individuals and organizations against cyber threats, identity theft, and fraudulent activities, InfoArmor developed sophisticated threat intelligence platforms, such as VigilanteATI, designed to monitor the dark web and other covert channels. To power these real-time, data-intensive operations, InfoArmor turned to MariaDB to handle massive data ingestion and high-performance querying.

Behind InfoArmor’s protection platform sits its Advanced Threat Intelligence division. InfoArmor employs human intelligence operatives who monitor closed dark web communities, underground marketplaces, and threat actor forums across Eastern Europe, Asia, South America, and the US. Coupled with automated web and forum scrapers, InfoArmor ingests massive volumes of compromised credentials and breach data to provide near real-time threat detection and alerting to its corporate subscribers.

The Challenge

As cyber breaches increased in both frequency and scale, such as a single spam bot leak containing over 700 million compromised credentials, InfoArmor’s threat intelligence platform faced exponential data growth. The core credential database rapidly expanded from 700 million records to over two billion rows.

However, the legacy database schema fragmented data across four separate tables: one for credentials and domains, one for encrypted passwords, one for regex matches, and a mapping table linking the IDs together. Querying this architecture required massive four-way joins across hundreds of millions of rows. Queries often triggered full table scans that took hours or days, causing 504 gateway timeouts on client dashboards and creating significant friction during live sales demos. Furthermore, application-level multithreading bugs caused connection leaks, while giant update transactions clogged binary logs >2GB and threatened database replication.

Why MariaDB?

To support its rapidly expanding threat repository without compromising high availability or search responsiveness, InfoArmor turned to MariaDB Enterprise Platform. The deciding factors were MariaDB’s deep expertise in optimizing high-volume transactional workloads and the ongoing operational support provided by the Remote DBA team. Working directly with MariaDB consultants, InfoArmor re-evaluated its core data structures to align database schemas directly with business utility. Furthermore, having MariaDB’s Remote DBA team manage off-site replication, configuration tuning, and real-time monitoring provided InfoArmor’s lean engineering team with 24/7 operational stability and peace of mind.

Schema Denormalization, Query Optimization, and Managed Replication

In collaboration with MariaDB experts, InfoArmor executed a comprehensive database modernization effort:

• Table Consolidation: InfoArmor eliminated the inefficient four-way join architecture by denormalizing and consolidating four segregated tables into a single primary table housing over two billion credential records along with associated passwords, domains, and source references.
  
• Aggregated Dashboards: Implemented meta-tables to pre-calculate high-level metrics across eCrime forum matches, credential watchlists, and network intelligence, enabling instant dashboard loads.
  
• Transaction and Connection Management: Resolved Python Pyramid ORM session leaks to prevent max-connection exhaustion, and restructured data ingest processes to process updates in smaller, controlled batches preventing binlog bloat and guaranteeing replication stability.
  
• Remote DBA Management: Integrated MariaDB Remote DBA specialists directly into InfoArmor’s team communications for proactive performance monitoring, schema advice, and replica management.

The Impact

Transitioning to MariaDB Enterprise Platform and streamlining the database schema transformed InfoArmor’s threat intelligence infrastructure to successfully manages and queries over 2 billion credential records and 40 million forum posts on a single consolidated table without query timeouts or performance degradation, resulting in Seamless Billions-Scale Performance.

System dashboard response times dropped from over 60 seconds to instant loading, eliminating 504 timeouts and allowing sales teams to conduct live platform demos seamlessly, leading to Elimination of Operational Bottlenecks.

Batching ingestion pipelines eliminated giant transaction spikes, protecting binary log limits and maintaining steady off-site replica synchronization, providing Replication & Binlog Stability.

Remote DBA support freed internal engineers from routine database maintenance, allowing them to focus on feature development, parser improvements, and threat intelligence analytics, ensuring enhanced engineering efficiency.

Future Outlook

As InfoArmor continues to expand its intelligence collection, the company expects to double its forum post ingestion to over 80 million records in the next 12 months. To support this growth, InfoArmor plans to evaluate MariaDB MaxScale to provide intelligent read/write splitting and automated query routing between master and replica nodes.

With MariaDB Enterprise and Remote DBA support as a foundation, InfoArmor is well-positioned to scale its threat detection capabilities and protect corporate employees against emerging global cyber threats.