This is 
id:wtatsuru, Assistant Producer of Mackerel.
Regarding Mackerel’s usage fee calculations from November 2016 to June 2022, there were incidents where users were charged more excessively than their actual usage due to mistakes in counting the number of hosts subject to billing in some organizations that stopped all host usage during the month.
This incident was confirmed after an inquiry from a customer on July 1st, 2022. After checking the situation, we fixed the billing system. We investigated the affected areas, including the past periods. With all the investigations complete, here is a report on the details. We deeply apologize for the inconvenience we caused due to this incident.
Incident Details
Mackerel’s usage fees are calculated based on the number of hosts used in a month.
The number of hosts used includes registered hosts on Mackerel and additional host count conversions when the limit for each plan is exceeded in some functions.
Handling of host conversion when plan limits are exceeded – Mackerel Support
Also, the number of hosts subject to billing is calculated as a monthly average.
Calculating the number of hosts – Mackerel Support
In this incident, when all host usage was stopped during the month, the average was taken from the period used instead of one month when calculating the number of exceeded hosts. As a result, users were billed a fee larger than actual usage.
Causes
The number of hosts used in Mackerel is logged in our database at fixed intervals. In this process for storing host counts, the total number of hosts when host metrics exceeded subscription limits was not logged in organizations with no host usage at all. In such cases, monthly averages should be calculated considering periods with no usage record. However, in this case, the average was taken only for periods when usage records existed, resulting in charges exceeding what is appropriate.
- (Correct Calculation) Number of units used = Total number of hosts used by time/the amount of time in one month
- (Wrong Calculation) Number of units used = Total number of hosts used by time/ the time with usage
Scope Affected
- Affected Period: November 2016 to June 2022
- Scope
- Of organizations that stopped host usage during a month (stopped all metric posts and check monitors), the number of hosts when the host metric figure exceeded the plan limit
Actions After the Discovery
- July 1st, 2022: the incident was discovered from a customer inquiry, and an investigation was started
- July 26th, 2022: The calculation system was updated so that the number of hosts is calculated correctly
- August 23rd, 2022: The scope affected in the past was fully investigated
- October 28th, 2022: This notice
Measures and Prevention of Reoccurrence
In response to this incident, we changed the algorithm for calculating the number of hosts when the number of host metrics exceeds the plan limit so that it would consider the usage period. We have reviewed all host number records/calculation logic of other billing items and have confirmed that similar incidents do not occur.
We have also added a step to confirm the proper application of the above changes when checking and processing the billing amount.
Usage Fee Refunds
We will refund the usage fees that may have been overcharged due to this incident. We will contact organization owners by email, so please check it. Customers who are not contacted individually were not affected.
The number of hosts subject to refunds has been calculated using the method below.
- Subject organizations are those with a usage history of paid plans during the corresponding period.
- For organizations that stopped all host usage during the month, we recalculated the number of host conversions due to metric excess, and the correct number of hosts was calculated
- The above amount and the number of hosts used to bill in the past were compared, and the number of units for refunds were calculated if there were differences.
Plans to Revise the Billing Process After Two Overcharging Incidents
After the incident we reported on May 31st, 2022, second overcharging incidents were discovered. We apologize for repeatedly inconveniencing our users.
Apology and Detailed Report on the Overcounting of Cloud Integration Metrics - Announcements
We take these incidents seriously and will implement comprehensive revisions of billing-related systems. The current billing processing system and the overall operations flow will be revised. Furthermore, we plan to compare host number counts and fee calculations, and add a system that can more comprehensively implement checks.
We will revise our systems and work to improve the quality of our services. We ask for your continuous support of Mackerel.
