Thank you for using Mackerel.
We deeply apologize for the inconvenience and concern caused by multiple intermittent incidents in Mackerel's labeled metrics feature between May 28, 2026, and June 23, 2026.
This report provides details regarding the overview of these incidents, their root causes, and our measures to prevent recurrence.
Background: Renewal of the Labeled Metrics Storage Infrastructure
Labeled metrics is a feature that allows users to store posted metrics along with their labels and use them for search, graph visualization, and monitoring (for details, please refer to the Help page).
With the growth of service usage, the conventional storage infrastructure was unable to scale sufficiently to handle the increasing volume of data, and it was anticipated that it would become unsustainable in terms of both performance and cost in the future. Therefore, we initiated an effort to rebuild the storage infrastructure into a new one capable of enduring large-scale data volumes while achieving fast search and short data reflection times.
The new infrastructure combines multiple data stores according to their respective roles, achieving higher scalability than before. However, this also increased overall system complexity. This series of incidents occurred during the migration process to this new infrastructure and during operations after the migration.
The migration was carried out in stages: writing to the new infrastructure began on March 16, 2026, and the complete switchover to the new infrastructure was finalized on April 6, 2026. However, during subsequent operations, the incidents described below occurred.
Overview and Causes of the Incidents
June 1, 2026: Issue Where Some Metrics Were Not Displayed on Graphs
Incident
An incident occurred where some labeled metrics were not saved or displayed correctly, resulting in failures when rendering query graphs. Additionally, query monitors referencing the affected metrics failed to function properly.
Cause
Immediately prior to the incident, there was an influx of metric submissions from a specific environment that significantly exceeded expectations. To protect the overall service, we introduced a rate-limiting mechanism. However, this mechanism contained a bug regarding histogram-type metrics. While histogram-type metrics are internally treated as multiple metrics upon submission, the metric count aggregation did not accurately reflect this number. As a result, rate limits were applied more strictly than intended, causing a portion of the label information that should have been saved to be missing. This issue occurred exclusively for customers submitting metrics that included histograms.
Response
We fixed the implementation bug in the rate limiter and improved the system so that metrics that should be saved are processed correctly. Additionally, as a temporary measure to enable graph display when searching for periods with missing labels, we expanded the scan range to periods without missing labels.
However, for some metrics whose processing was skipped during the period of the bug, the data was not saved and could not be recovered.
Due to the above, data containing "histograms" posted between May 28, 2026, 15:53 and June 1, 2026, 18:52 JST was lost, resulting in overcharges for some customers. We are currently issuing refunds.
June 12, 2026: Issue Where Past Metrics Prior to a Certain Time Were Not Displayed
Incident
An issue occurred in labeled metrics where past metrics were not displayed for certain display granularities.
Cause
During the fix for the June 1 incident, we added a temporary process to expand the search scan range into the past when searching within the affected period. However, there was a bug in how that range was specified. When attempting to display periods prior to around 9:00 AM on May 27, 2026, an unintended range was scanned, causing metrics for that period to not be displayed.
Response
We fixed the bug in the process that backtracks the reference range.
Note that no data was lost due to this bug, and with the fix completed, past metrics have fully recovered to a normal display state.
June 17, 2026: Issue Where Metrics Without service.name Attribute Could Not Be Viewed
Incident
An issue occurred where some labeled metrics without the service.name attribute became unavailable. Specifically, they could not be displayed in Metric Explorer or dashboards, and query monitors referencing those metrics did not function.
Cause
Under the old infrastructure, metrics submitted without a service.name attribute were automatically assigned a fallback attribute value. This fallback handling was inadvertently omitted during the migration to the new infrastructure.
Response
We sequentially applied the attribute value unknown_service to metrics that lacked the service.name attribute, and completed remediation for all affected metrics on June 23, 2026.
Recurrence Prevention Measures
We believe there were two root causes underlying this series of incidents:
- Insufficient regression testing: There was a lack of validation to guarantee that the new infrastructure behaved in the exact same manner as the old infrastructure. Despite rebuilding the platform, we did not sufficiently verify whether legacy behaviors were preserved.
- Incomplete understanding of failure scenarios in a complex system: In the new infrastructure combining multiple data stores, we did not fully comprehend what kinds of anomalies would lead to critical failures. Despite the increased complexity of the architecture, validation against failure scenarios lagged behind. Furthermore, this complexity made it difficult to grasp the scope of impact when making code modifications, which contributed to the June 12 incident.
Based on these reflections, we have already initiated the following measures and will continue to execute them going forward:
- Regression test suite expansion: We will establish comprehensive regression tests to guarantee identical behavior between the old and new platforms.
- Failure scenario validation: Drawing on insights gained from this series of events, we will identify additional anomaly scenarios and strengthen our testing regimen. In particular, we will focus on verifying how anomalies in situations involving multiple data stores impact our customers.
- Architecture consolidation: We will consolidate multi-datastore write logic as much as possible to transition toward an architecture that maintains data consistency more easily. We will review the core system structure to prevent distributed processing from becoming a breeding ground for bugs.
- Review process enhancements: To make it easier to understand the scope of impact and identify inconsistencies during code modifications, we will introduce AI-assisted review tools and improve our overall code review process.
Improvements to Information Communication
In addition to incident prevention measures, we have made the following improvements so that customers can quickly assess the situation when an issue occurs:
Explicit Impact Scope in Web Console Banners
During maintenance and incidents, Web console notifications were previously limited to abstract messages such as "Currently, some systems of Mackerel are experiemcing trouble." We have updated these to include explicit impact scopes, such as "Currently, some systems of Mackerel (posting, querying, and viewing labeled metrics) are experiencing trouble." (Release announcement: Banner messages during maintenance and incidents now indicate the affected functional scope). This allows you to immediately evaluate the impact on the features you are using.
Addition of "Labeled Metrics" Component to Status Page
We have also improved the component structure of our official status page (https://status.mackerel.io). Previously, metrics ingested via agents/cloud integrations and labeled metrics were both handled under a single "Metrics" component. We have now added a standalone "Labeled Metrics" component. Going forward, when maintenance or incidents specifically affect labeled metrics, we can communicate the impact scope much more clearly.
We sincerely apologize once again for the inconvenience caused to our customers by these multiple incidents in the labeled metrics feature.
Thank you for your continued support of Mackerel.
