How to make "page template" admin column sortable?

I'm using the following code to add a "Template" column into the Pages admin columns:

// Manage the columns shown
add_filter( 'manage_pages_columns', 'add_template_col' );
function add_template_col( $columns ) {
    $columns['page-layout'] = 'Template';
    return $columns;
}

// Populate non-standard columns
add_action( 'manage_pages_custom_column', 'fill_template_col', 10, 2);
function fill_template_col( $column, $post_id ) {
    if ( 'page-layout' === $column ) {
        $set_template = get_post_meta( $post_id, '_wp_page_template', true );
        if ( ! $set_template ) {
            _e( 'n/a' );  
        } elseif ( $set_template == 'default' ) {
            echo 'Page';
        }
        $templates = get_page_templates();
        ksort( $templates );
        foreach ( array_keys( $templates ) as $template ) :
            if ( $set_template == $templates[$template] ) echo $template;
        endforeach;
    }
}

This is working, but I'd like users to be able to sort by this column. I've tried various implementations of admin column sorting solutions, which work for sorting other columns (for example, adding and sorting columns based on Advanced Custom Fields, etc.), but nothing I've tried seems to work for sorting the Templates column I've created. Here's and example what I've tried, but it's not working:

// Add to the list of sortable columns
add_filter( 'manage_edit-pages_sortable_columns', 'template_sortable_columns');
function smashing_realestate_sortable_columns( $columns ) {
  $columns['page-layout'] = '_wp_page_template';
  return $columns;
}

I've also tried hooking into the query (which is what I do when columns are based on custom fields), but this doesn't help either:

// Alter query
add_action( 'pre_get_posts', 'template_col_orderby' );
function template_col_orderby( $query ) {
    if( ! is_admin() || ! $query->is_main_query() ) {
        return;
    }
    if ( '_wp_page_template' === $query->get( 'orderby') ) {
        $query->set( 'orderby', 'meta_value' );
        $query->set( 'meta_key', '_wp_page_template' );
        $query->set( 'meta_type', 'char' );
    }
}

Obviously I'm doing something wrong here and would really appreciate some advice.