Mechasmmechasm logoMechasmbeta
Back to Docs

Advanced Features

Explore advanced reporting features including Top Lists, Reliability Metrics, and Time Windows.

The Reports dashboard includes powerful advanced features for deep analysis and comprehensive test insights.

Time Window Filtering

Control the time scope of your analysis with flexible time window options.

Available Time Windows

  • 7 Days: Recent short-term trends and immediate issues
  • 14 Days: Medium-term patterns and weekly cycles (default)
  • 30 Days: Long-term trends and monthly analysis

How It Works

The time window selector appears in the top-right corner of the Reports dashboard and affects:

  • Team Overview: Filters tests created within the selected timeframe
  • Project Metrics: Scopes analysis to the chosen time period
  • Test Insights: Applies temporal filtering to tag-based analysis

Use Cases

7-Day Window: Ideal for sprint retrospectives, immediate issue triage, and recent deployment impact analysis.

14-Day Window: Perfect for bi-weekly reviews, identifying emerging patterns, and medium-term quality trends.

30-Day Window: Best for monthly reports, long-term trend analysis, and comprehensive quality assessments.

Auto Refresh

Keep the dashboard live without manual reloads.

  • Options: Off, Every 1 min, Every 5 min
  • Placement: Top-right controls (above Time Window)
  • Persistence: Saved in localStorage per user
  • Behavior: Uses live data and avoids refetch when tab is hidden

Top Lists

Quickly identify your most problematic and highest-impact tests across three critical dimensions.

Slowest Tests

  • Metric: Execution duration in seconds
  • Ranking: Ordered by longest execution time
  • Use Case: Performance optimization and resource planning
  • Action Items: Consider test splitting, environment optimization, or infrastructure scaling

Flakiest Tests

  • Calculation: Based on step failures, total failures, and error analysis
  • Ranking: Weighted score combining multiple failure indicators
  • Use Case: Reliability improvement and maintenance prioritization
  • Action Items: Review test logic, environment dependencies, and timing issues

Most Failing Tests

  • Definition: Tests with the highest total failure counts over the selected window
  • Data Source: Aggregated totalFailCount from recent runs
  • Use Case: Quality triage and critical issue identification
  • Action Items: Investigate root causes, fix underlying issues, or quarantine problematic tests

Visual Indicators

Each test in the Top Lists shows:

  • Status Badge: Current execution state with color coding
  • Metric Value: Specific measurement (duration, issues, failures)
  • Enabled State: Indicates if test is active or disabled

Reliability Metrics

Advanced analysis of test stability, maintenance needs, and execution environment patterns.

Retry & Quarantine Analysis

Tests with Retries: Count and percentage of tests experiencing step or total failures, indicating potential stability issues.

Average Step Failures: Mean number of step failures per test, helping identify systemic issues vs isolated problems.

Most Failing Tests: Tests with the highest total failure counts over the selected time window

Quarantined Tests: Disabled tests that require maintenance or investigation.

Reliability Score: Calculated as (100 - retry rate), providing an overall stability metric with visual indicators.

Alert System: Automatic warnings when retry rates exceed 20%, suggesting environment or test stability issues.

Data Quality

Understand the completeness and reliability of your test data.

Overall Quality Score: Percentage of tests with complete key fields (duration and timestamps). Shown prominently with color status.

Missing Duration: Count of tests without recorded execution time and its impact percentage on overall metrics.

Missing Timestamps: Count of tests missing start/end times and its impact percentage on freshness/recency calculations.

Alerting: A warning banner is displayed when the quality score falls below 90%, recommending data collection and execution pipeline review.

Location: Appears in the Reliability Metrics section alongside Retry & Quarantine.

Data Quality Indicators

The Reports dashboard includes built-in data quality assessment to help you understand the completeness and reliability of your metrics.

Missing Data Detection

Duration Gaps: Identifies tests without recorded execution times, affecting performance analysis accuracy.

Timestamp Issues: Flags tests missing start/end times, impacting freshness and recency calculations.

Impact Assessment: Shows percentage of affected tests and how missing data influences metric reliability.

Quality Scoring

Visual indicators show the completeness of your test data (aligned with the dashboard):

  • Green: Excellent data (≥ 90% coverage)
  • Amber: Good data (≥ 70% coverage)
  • Red: Incomplete data (< 70% coverage)

This transparency helps you make informed decisions about metric interpretation and data collection improvements.

Advanced Reports Reliability

Best Practices

Time Window Selection

  • Use 7-day windows for immediate issue response and sprint analysis
  • Apply 14-day windows for pattern recognition and trend validation
  • Leverage 30-day windows for comprehensive reporting and long-term planning

Top Lists Analysis

  • Review Slowest Tests weekly to maintain performance standards
  • Monitor Flakiest Tests daily during active development periods
  • Prioritize Most Failing Tests for immediate investigation and fixes

Reliability Monitoring

  • Set alerts when reliability scores drop below 80%
  • Investigate environment-specific failures for infrastructure issues
  • Use quarantine metrics to track maintenance backlog and technical debt

The advanced features transform the Reports dashboard from a monitoring tool into a comprehensive quality intelligence platform, enabling data-driven decisions and proactive quality management.