Technical appendix for "Cart challenges, empirical methods, and effectiveness of judicial review"
Mikołaj Barczentewicz (www.barczentewicz.com)
[Last updated on 30 August 2021]
This appendix consists of two parts:
Part A: Case level data on judicial review in the High Court¶
Ministry of Justice (‘MoJ’) publishes a very helpful judicial review case level dataset (‘JR case level dataset’) as a part of their ‘Civil justice statistics quarterly’. This dataset is available in the form of a CSV file that contains information about the fate of individual claims for judicial review issued in the High Court since 2000 (the ‘JR_csv.csv’ file from the ‘Civil Justice and Judicial Review data (zip file)’ collection). Importantly, each judicial review claim is only counted once in that dataset and the ‘Year’ column represents the year the claim was issued, irrespective of when it was closed. The JR case level dataset includes ‘Cart – Immigration’ and ‘Cart – Other’ among the topics to which each case is assigned, which allows for separate analysis of Cart and non-Cart judicial review claims. One of the main downsides of that dataset is that it does not contain information on which claims are ‘withdrawn by consent’ (or settled) before a substantive hearing, rendering more difficult the task of studying the rates of settlement in judicial review.
MoJ also publishes a Guide to the statistics. The Guide doesn't answer the question about the Year field - but by comparing the CSV with numbers from the MoJ spreadsheet Civil_Justice_Statistics_Quarterly_October_to_December_2020_Tables.ods, I can tell that Year represents the year when the case was lodged.
Code samples¶
The following code samples illustrate how I queried that dataset (using Python and Pandas).
import pandas as pd
jr_csv_df = pd.read_csv('../MoJ_statistics_JR/workload_csv/JR_csv.csv')
Total numbers of Cart claims brought annually¶
total_cart_per_year = jr_csv_df[
(jr_csv_df.Topic.isin(['Cart - Immigration', 'Cart - Other']))
].groupby(['Year']).Year.count().rename('total_cart_per_year')
total_cart_per_year
Numbers of Cart claims as percentage of all judicial review claims annually¶
Total numbers of all JR claims annually (from 2012):
total_jr_per_year = jr_csv_df[
(jr_csv_df.Year>=2012)
].groupby(['Year']).Year.count().rename('total_jr_per_year')
total_jr_per_year
Calculate ratios:
total_per_year_df = pd.DataFrame([total_cart_per_year, total_jr_per_year]).T
total_per_year_df['cart_ratio_total'] = total_per_year_df.apply(lambda row: row.total_cart_per_year/row.total_jr_per_year, axis=1)
total_per_year_df.style.format({
"cart_ratio_total": "{:.2%}",
})
What percentage of claims that reach the permission/renewal stage are Cart claims?¶
Cart claims annually:
cart_at_permission_per_year = jr_csv_df[
(jr_csv_df.Topic.isin(['Cart - Immigration', 'Cart - Other'])) &
(
(jr_csv_df.permission == 1) |
(jr_csv_df.renewal == 1)
)
].groupby(['Year']).Year.count().rename('cart_at_permission_per_year')
cart_at_permission_per_year
All claims annually:
at_permission_per_year = jr_csv_df[
(jr_csv_df.Year>=2012) &
(
(jr_csv_df.permission == 1) |
(jr_csv_df.renewal == 1)
)
].groupby(['Year']).Year.count().rename('at_permission_per_year')
at_permission_per_year
Calculate the ratio:
at_permission_per_year_df = pd.DataFrame([cart_at_permission_per_year, at_permission_per_year]).T
at_permission_per_year_df['cart_ratio_at_permission'] = at_permission_per_year_df.apply(lambda row: row.cart_at_permission_per_year/row.at_permission_per_year, axis=1)
at_permission_per_year_df.style.format({
"cart_ratio": "{:.2%}",
})
Compare the last two aggregate tables¶
cart_ratios_df = at_permission_per_year_df.join(
total_per_year_df,
on='Year'
)
cart_ratios_df[['cart_ratio_at_permission', 'cart_ratio_total']].style.format({
"cart_ratio_at_permission": "{:.2%}", "cart_ratio_total": "{:.2%}",
})
cart_ratios_df[cart_ratios_df.index>=2014].sum()
cart_ratios_df[cart_ratios_df.index>=2014].mean()
Compare numbers of Cart claims with other kinds of claims¶
all_topics_agg_df = pd.DataFrame(jr_csv_df[
jr_csv_df.Year>=2014
] \
.groupby(['Topic']).Year.count() \
.rename('cases'))
total_claims = len(jr_csv_df[
jr_csv_df.Year>=2014
])
all_topics_agg_df['percent_of_all'] = all_topics_agg_df.apply(lambda row: row.cases/total_claims, axis=1)
all_topics_agg_df.sort_values('cases', ascending=False).head(20).style.format({
"percent_of_all": "{:.2%}",
})
all_topics_years_df = pd.DataFrame(jr_csv_df[
jr_csv_df.Year>=2018
] \
.groupby(['Year', 'Topic']).Year.count() \
.rename('cases'))
all_topics_years_df[all_topics_years_df.cases>120] \
.sort_values(['Year','cases'], ascending=False)
Part B: Study of Upper Tribunal decisions¶
The following overview covers only my analysis of texts of decisions of the Upper Tribunal. However, the full paper relies also on a separate analysis of
The goal of my empirical study of texts of decisions of the Upper Tribunal was twofold:
- to identify decisions of the Upper Tribunal which followed a successful Cart judicial review ('Cart JR', 'CJR'),
- to identify which of those decisions resulted in setting aside of the appealed decision of the First-Tier Tribunal ('FTT') - in other words: which decisions resulted in a 'positive result' of a Cart JR according to the criteria set by the Indeptendent Review of Administrative Law.
This study involved the following stages:
- Collecting the data on all Upper Tribunal decisions available in various subpages within
gov.ukand then creating one database of all those decisions (over 42,000 decisions). I also created a custom search engine interface for the database. - Designing and testing an intentionally overinclusive search query to identify all decisions potentially following successful Cart JRs.
- First manual classification of all decisions returned by the search query from (2) to remove the obviously irrelevant ones.
- Second manual classification of decisions remaining after (3): coding of 26 parameters (variables) into a separate dataset (most importantly: whether the appealed FTT decision was set aside).
I discuss Stage 1 (the dataset) after Stages 2 and 3-4.
Stage 2: Searching for Cart decisions¶
My search engine interface¶
I created a custom search engine UI for the Elastisearch database with Vue.js based on vue-search-ui-demo.
Search query¶
After some trial and error, I settled on the following query (using Elasticsearch's implementation of Lucene query syntax):
("refusal permission appeal quashed "~30) OR ("refuse permission appeal quashed "~30) OR ("cart" NOT "cart horse"~10) OR ("54.7A") OR ("judicial review refusal permission"~30) OR ("judicial review refuse permission"~30) OR ("judicial review refused permission"~30) OR ("Upper Tribunal refuse permission"~3) OR ("Upper Tribunal refused permission"~3) OR ("Upper Tribunal refusal permission"~3)
In other words, my query is a disjunction of the following queries:
- words
refusal permission appeal quashedwithin the edit distance of 30 or- the same with following sets of words:
refuse permission appeal quashed,judicial review refusal permission,judicial review refuse permission,judicial review refused permission
- the same with following sets of words:
- word 'cart' (not case-sensitive because the database conforms keywords to lowercase to allow faster processing), but NOT within the edit distance of 10 from any appearance of word 'horse' or
- phrase '54.7A' or
- words
Upper Tribunal refuse permissionwithin the edit distance of 3 or- the same with words
Upper Tribunal refused permission,Upper Tribunal refusal permission.
- the same with words
This query is meant to be overinclusive and it was necessary to read the decisions (see Stages 3-4) to see which of them are really relevant.
I also limited the query only to decisions that came after the Supreme Court's judgment in Cart, although that is likely slightly overinclusive:
- R. (on the application of Cart) v Upper Tribunal [2011] UKSC 28; [2012] 1 A.C. 663 was decided on 22 June 2011.
- CPR 54.7A was added by The Civil Procedure (Amendment No.2) Rules 2012, which entered into force on 1 October 2012.
The numbers of cases identified through this query in each of my datasets are:¶
- Immigration and Asylum Chamber: 548 (adding to the query above, I also removed all cases with identifiers starting with the letters "JR" to exclude judicial review cases heard in the Upper Tribunal)
- Administrative Appeals Chamber
- since 2016: 36
- before 2016: 60
- Tax and Chancery Chamber: 31
- Lands Chamber: 17
Stages 3 and 4: Manual classification¶
Results after manual classification (numbers of likely UT follow-ups on successful CJRs)¶
The main fashion in which the query above was overinclusive was in identifying (non-Cart) judicial review cases or references to such judicial review cases. I manually checked all positive results of the above query (with the exception of pre-2017 cases in the Immigration and Asylum Chamber) and identified the following numbers of likely follow-ups on successful CJRs:
- Immigration and Asylum Chamber
- manually classified for 2017-2020: 116 (314 before manual classification)
- before 2017 (not manually classified): 234
- Administrative Appeals Chamber
- since 2016: 5
- before 2016: 1
- Tax and Chancery Chamber: 4
- Lands Chamber: 7
The numbers after manual classification should not be treated as the total number of UT decisions following-up successful Cart JRs because there are gaps in coverage of UT decisions published within gov.uk that increase from 2016 backwards. See my estimate of comprehensiveness of the Immigration and Asylum Chamber (UTIAC) dataset below.
The final dataset for 2017-2020¶
The file UT_cart_cases_2017-2020.csv attached to this paper contains the results of the final manual classification (coding).
Regarding the cart_application_year column, note that post-Cart judicial review decisions of the Upper Tribunal are not necessarily promulgated in the same year in which the Cart application is filed in the High Court. I adjusted for this using a complex formula taking as inputs all information about the Cart claim I was able to ascertain from the text of the UT decision (sometimes a date of the Cart judicial review application was mentioned, more often the date the Cart permission or the Cart quashing took place, sometimes none of those dates), as well as statistics on average timeliness between various stages of the process.
Note also that this dataset contains 6 Scottish (Eba) cases that are classified as 'Cart' cases in MoJ JR case level dataset, but did not originate in the High Court. I did not include them in the calculations I used in the paper, but I include them in the file for completeness.
Stage 1: The dataset of Upper Tribunal decisions¶
The Upper Tribunal has four chambers and decisions of each of the chambers are available from different sources (in the .gov.uk domain). To create a dataset of available decisions of the Upper Tribunal I scraped data from five databases in the gov.uk domain. Some of the decisions are available through the gov.uk API, but most aren't (including the decisions of the Immigration and Asylum Chamber).
Immigration and Asylum Chamber (UTIAC)¶
There are 33,810 UTIAC decisions listed on the government's website. For 22,779 of those, texts of UTIAC decisions are available on individual pages of decisions (eg here), but for the remaining 11,031 one must download a DOC(X) or PDF file linked on the decision page.
Using the Python library Scrapy, I downloaded the HTML files of pages of individual UTIAC decisions. I also downloaded 11,027 DOC, DOCX, and PDF files of texts of decisions where they were not included in HTML pages (4 documents were corrupted or inaccessible). I then converted the PDF files (using Adobe Acrobat) and the DOC/DOCX files (using DEVONthink 3) to HTML.
I then combined 33,806 texts of decisions, together with some available meta-data, into one dataset in an Elasticsearch database which allows for convenient complex searches of large datasets.
The following figure shows how many decisions decided in each year I collected.