Three of the court's conservatives said the court should have taken up the case, but Justice Kavanaugh and Chief Justice Roberts voted with the majority.