AOC

Challenge

How can we rigorously measure citizen satisfaction in the use of digital public services.

Problematic

From the AOC Consortium, we promote the development of indicators that enable policymakers and public executives to know the social return of digital administration and open government initiatives, and to evaluate systematically and rigorously their effectiveness and efficiency. so that policymakers and public executives can decide on priorities and make the best decisions.

It is very important to evaluate the perception of citizens' satisfaction.

As the ultimate goal of public administration is to provide services that respond to the needs and expectations of citizens, it is very important to evaluate their perception of satisfaction.

But does anyone measure it?

The AOC Consortium has studied how citizen satisfaction with public services in general (not restricted to the digital level) is measured to governments around the world. He also contacted the reference bodies on evaluation issues: iValua, Barcelona City Council, Barcelona Provincial Council, the State Agency for the Evaluation of Public Policies and the Quality of Services (no longer exists today) and international websites.

The conclusions were disappointing:

  • Generally speaking, very few public administrations measure citizen satisfaction and even less publish their results openly.
  • There are different methodologies for assessing citizen satisfaction which are used mainly in the private sector. We have not been able to find a methodology that could be considered a standard by the public sector.
  • The Diputación de Barcelona made a very good report for its town councils, on demand, called "Image and balance of the municipal management". But she has stopped doing it for some time.
  • The Barcelona City Council conducts excellent citizen satisfaction studies using the methodology of expectations evaluation. This method is very advanced but very complex and expensive to use by the AOC Consortium because it involves measuring the difference in expectations before using the public service and the subsequent result.
  • Surveys with generic satisfaction questions give results that are not easily interpretable or comparable and have little value in themselves. For example, getting a score of 7 (out of 10) can be good in some contexts or question types, and bad in others. Also, the answers are often made in a frivolous and unreflective way.
  • In the private sector it was detected that the main methodology used by the leading companies in the world is the Net Promoter Score (NPS).

NPS to the rescue

There are numerous studies that have proven that the Net Promoter Score (NPS) measures word-of-mouth very well based on a specific question. "How likely are you to recommend this service to a friend or friend?". This question has the psychological effect of asking for a recommendation and we instinctively people are more rigorous in the answer because we don't like to make a bad recommendation to someone close.

On a scale of 0 to 10, the following are considered:

  • Promoters: those who value 9 or 10 are very satisfied with the service or product and will speak enthusiastically and proactively.
  • Liabilities: those who value a 7 or 8 are satisfied with the service or product and will speak positively if asked but will not enthusiastically do so.
  • Detractors: those who value 0 to 6 are dissatisfied with the service or product and will speak negatively

Studies show a high correlation between these results and subsequent verbal feedback. The AOC has validated that ratings have a high correlation with the written comments we ask users in the digital survey.

In the private sector, NPS results are used as a measure of customer loyalty and future purchases. Promoters are loyal customers, and detractors are customers who will never buy again. In the public sector, there is no point in talking about customer loyalty because public services are, in general, "monopolistic" and there is no competition between public administrations.

In 2016, however, the study "Adapting Net Promoter thinking in public sector organizations" was published by Juho Hakola, from the University of Jyväskylä (Finland), which explains how the NPS methodology can be applied to the public sector.

Applied solution

Use the NPS methodology adapted to the public sector according to the study “Adapting Net Promoter thinking in public sector organizations"By Juho Hakola, to evaluate the satisfaction of citizens in the use of digital public services.

This adaptation is very easy to understand and informs us of the% of citizens who:

  • They are very satisfied (9-10) and will speak very well of the service
  • Satisfied (8-9) and will speak well of the service
  • Dissatisfied (0-6) and they will talk bad about the service

This methodology has been implemented for both citizens and public employees using the AOC services. It has been applied in a context that has greatly facilitated the high volume of surveys and to ensure that there is a sufficiently representative sample. Specifically, we are collecting more than 30.000 monthly surveys from all digital services.

Criteria for collecting the answers

  • Satisfaction surveys are presented at a "hot" time in the process and immediately. For example, with the paperwork, the survey is presented at the end of the paperwork.
  • The surveys are presented with a pop-up window that calls for participation.
  • The surveys are very brief. They have an average duration of less than 1 minute and generally have a few very selected questions:
    1. NPS where the value of the public service recommendation is 0 to 10
    2. Open question why of the valuation
    3. Ask if you want to make a proposal for improvement or suggestion
  • Surveys are non-invasive. Users who respond to a survey for a particular service will not be re-submitted after a few months.

With this methodology we have managed to collect important quantitative information on citizen satisfaction with digital public services. Now, we still have challenges to solve…

Pending challenges

  1. Analyze the huge volume of comments collected so far
    In addition to the recommendation between 0 and 10, we have approximately 10.000 comments per month on the assessment made by users that can give us important qualitative information but that we cannot analyze with human resources. We have done an analysis pilot with natural language interpretation algorithms to analyze eNotum service surveys with satisfactory results and we need to systematize and extrapolate it to other services.
  2. We do not have the whole "photograph"
    Satisfaction surveys are only available when the process is complete. We are not collecting ratings from users who have started or failed an AOC service. We have yet to explore how we can detect these cases because it is not easy to know when they are caused by a service problem or because the user voluntarily withdraws.

Status of the project

  • In production and applying to all digital services of the AOC Consortium.
  • We also carry out satisfaction surveys for those attending the AOC Conferences and other events in which we collaborate.