MPSC workgroup: Investigating the relationships of different metrics

What are the properties of a good metric?

Broadly, a good metric for regulatory review (1) measures a clinically important outcome, and (2) does not clearly depend on factors outside the control of the program. For example, the deceased donor transplant rate ratio measures a clinically important outcome, i.e., getting patients transplanted, but depends on the relative donor supply and demand around the transplant program. Thus, the deceased donor transplant rate ratio is not a good metric for regulatory review. In contrast, the offer acceptance rate ratio also measures the propensity of transplant but does not clearly depend on donor supply and demand. Thus, the offer acceptance rate ratio may be good metric for regulatory review.

What are good metrics for a dashboard?

A dashboard is a collection of good metrics for measuring clinically important but different outcomes at a transplant program. Transplant professionals are best positioned to assess the clinical importance of different outcomes. In contrast, the relationship among different metrics can be assessed visually and statistically with, respectively, scatterplots and correlation coefficients. Good combinations of metrics will have scatterplots without an obvious pattern (e.g., a circular splatter), while poor combinations of metrics will have scatterplots with obvious patterns (e.g., a cloud of points around a line). The former situation corresponds to a correlation coefficient of approximately 0, while the latter combination will have a strong correlation coefficient, e.g., close to or higher than 0.5.

How can we use this application?

This application allows users to compare potentially good metrics currently available in the program-specific reports across kidney, liver, lung, and heart programs. Specifically, the user selects the options from the four dropdown menus (see Figure 1), and then the application generates a scatterplot.


Figure 1. The different options available to the user.

As noted above, good metrics for a dashboard will capture different program outcomes and therefore have a relatively low correlation. For an example of good metrics, Figure 2 (see below) presents the scatterplot and correlation coefficient between the waitlist mortality rate ratio and the offer acceptance rates ratio.


Figure 2. An example of two metrics with a minimal correlation and therefore capturing different outcomes. Thus, a dashboard could include both of these metrics.


In contrast, a bad combination of metrics will capture similar outcomes and therefore have a relatively high correlation. As an example of a bad combination of metrics, Figure 3 (see below) presents the relationship between the hazard ratios for 1-month and 1-year graft survival.


Figure 3. An example of two metrics with a strong correlation and therefore capturing similar outcomes. Thus, a dashboard would not need to include both of these metrics.