As noted in the text of the Methodology, there are six different assessment tools,
namely desk-based literature reviews, other desk-based reviews, key informant
interviews, self-assessments, office visits and RTI request testing. This planning
document aims to help those running the assessment to plan the application of these
different tools, given their cross-cutting use in different assessment areas (Central
Measures, Public Bodies: Institutional Measures, Public Bodies: Proactive Disclosure
and Public Bodies: Reactive Disclosure).
a) Desk-based literature review
Desk-based literature review is used for all four assessment areas. Given the nature
of this work, it would make sense to have one actor do all of the desk-based
literature review work. This is because information relevant to various assessment
areas may be found in the same document (such as the annual reports of the
oversight body and public authorities). It would be very inefficient for different
actors to go through the same documents looking for different information. At the
same time, this will require the actor to have a strong overview of all of the
substantive issues that are being assessed, so that all of the relevant information in
the literature is captured.
Some of the key documents to review here include:
- The annual reports of the oversight body and the public authorities which are being reviewed
- Any other reports issued by the oversight body and the public authorities which are being reviewed
- Reports by civil society organisations, whether local or international, about implementation, including any testing they may have done (for example by making RTI requests)
- Any relevant reports by inter-governmental organisations
- Any relevant official reports which may have been issued (i.e. by the government)
- Relevant media reports, including blogs
- Relevant records of parliamentary discussions
b) Other desk-based reviews
There are two ‘other desk-based reviews’. The first, in the Central Measures
assessment area, is a desk review of the decisions on appeals. These decisions
should be available on the oversight body’s website. If not, they may be available
directly from the oversight body (including, where necessary, through making an
RTI request for them). In many cases, oversight bodies decide a great many appeals
every year. In this case, it may not be realistic to review them all. Instead, it might
make sense to review a random selection, say one in ten or twenty decisions.
Decisions should also be reviewed over a period of years.
The key types of information that are being assessed through this review are:
- how long does it take to process appeals (on average, longest 10%)?
- do the decisions suggest that the system is geographically accessible (i.e. are they coming from different parts of the jurisdiction)?
- are appropriate decisions being made (based on what the facts of the case suggest would be an appropriate outcome)?
- are appropriate remedies being awarded (again based on what is allowed and what seems appropriate based on the facts of each case)?
- do the decisions suggest that appropriate due process protections have been respected (for example because it is clear that both parties have been given a chance to make representations)?
The second ‘other desk-based review’, in the Proactive Publication assessment area,
is a desk review of what has been published by public authorities on their websites.
This is the primary assessment tool for this area.
The key types of information that are being assessed through this review are:
- are all of the documents that the law requires being disclosed proactively? Does the public authority interpret the legal requirements narrowly or more generously?
- is it reasonably easy to find these documents?
- are the documents available in different languages?
- are disclosures and the websites containing them SCAG 2.0 compliant?
- has any complex information been ‘translated’ into simpler language so that citizens can understand it?
c) Key informant interviews
This is a key assessment tool which is provided for in relation to every single
assessment area. But different types of experts (officials, civil society
representatives, etc.) need to be interviewed for different assessment areas. Given
that it does not make sense for different actors to interview the same experts (i.e. to
ask experts to participate in more than one interview), this is again an area where it
would make sense for one actor to do all of the interviews (or at least all with one
category of expert).
Breaking the assessment areas down in terms of the type of expert gives the
following picture:
| Type of Expert | Assessment Areas |
| Members of the oversight body | Central Measures |
| PIOs | Institutional Measures, Proactive Disclosure, Reactive Disclosure |
| Senior officials | Central Measures, Institutional Measures, Proactive Disclosure, Reactive Disclosure |
| Other officials (IT staff) | Proactive Disclosure |
| Civil society representatives | Central Measures, Institutional Measures, Proactive Disclosure, Reactive Disclosure |
| Key media users | Central Measures, Institutional Measures, Reactive Disclosure |
| Complainants | Central Measures |
It may be difficult to identify some of these experts, such as key media users, other
key requesters and complainants but various techniques can be used to try to get
around this problem.
Sample key informant interview questionnaires for each type of expert are provided
as part of the Comprehensive Methodology.
d) Self-assessments
This is another assessment tool which is provided for in relation to every single
assessment area. The breakdown of who needs to undertake the self-assessment is
as follows:
- Central Measures: Oversight body
- Institutional Measures: Public Authorities
- Proactive Disclosure: Public Authorities
- Reactive Disclosure: Public Authorities
Only one self-assessment exercise should be directed at public authorities, which
should cover the three relevant assessment areas. In most cases, it will make sense
to provide the self-assessment to the PIO and then let him or her decide, in
consultation with others at the public authority, how it will be done. For the
oversight body, the self-assessment should probably be provided to the chief
commissioner.
Note that the questions in these self-assessment questionnaires are very similar to
the KII questions for members of the oversight body and IOs/senior officials. If this
self-assessment is likely to go to the same person again (instead of another person
at the oversight body or public authority), it might make sense just to do one or the
other (i.e. either the KII or the self-assessment but not both).
Sample self-assessment questionnaires for both the oversight body and individual
public authorities are provided as part of the Comprehensive Methodology.
e) Office visits This assessment tool is designed to assess whether information is being published
proactively in physical forms at the office (such as on a notice board). But it might
make sense to schedule any key informant interviews with staff there at the same
time and perhaps also the self-assessments.
f) RTI request testing
This is a complex and sophisticated assessment tool for which an entire protocol
and reporting format has been developed (see the relevant section of the
Comprehensive Methodology). The requesting exercise takes by far the longest of all
of the assessment tools, so it should be started first.