The IMD, a key tool for evaluation and continuous improvement

Challenge

As we can measure the degree of digital maturity of an entity and a territory so that almost any administration can accelerate its transformation to improve people's lives?

Problematic 

The digital transformation of an organization is a complex process that requires a constant evaluation of the progress made. However, developing an instrument to measure the degree of digital maturity involves a series of difficulties:

• there is no precise definition of the concept of digital government that unequivocally determines its dimensions and the indicators that define it;
• the processes of collection and integration of the data necessary to develop these indicators present difficulties;
• sufficient or quality data is not always available, which leads to having to discard some indicators from the analysis;
• it must meet the needs of all councils equally and allow comparisons to be made, regardless of their size. This is not trivial, since the population conditions the set of services to be provided and the potential volume of activity of the digital services it provides.

In addition to the complexity involved in measuring the degree of digital transformation of an entity, the fact that most municipalities in Catalonia are small or medium-sized and have all kinds of limitations to face this challenge, such as:

1. very limited financial resources;
2. data access difficulties;
3. lack of capabilities on how to define and generate digital key indicators;
4. impossibility of being able to make comparisons with other similar administrations due to difficulty in accessing the data and lack of data standards; i
5. public workers without advanced analytical skills.

Local administrations therefore need a "turnkey" solution that allows them to evaluate their change process with minimal effort and make the most correct decisions possible based on data. To respond to this need, the AOC has created the Digital Maturity Index (from now on, IMD), a tool that assesses the degree of digital transformation of Catalan local administrations and allows administrations to compare themselves between them

Proposed solution

The IMD is a tool that provides us knowledge to design a digital transformation strategy for all the APCs, as well as plan the necessary actions that will need to be carried out.

It's not just another indicator. It is an information system (“relates indicators”) that allows:

• Know the degree of digitization of a public body (strong points and weak points of each body) and understand the progress of its transformation process.
• Compare and gain perspective. The digital maturity of each of us becomes more evident when it is compared (contrasted) with others. In addition, it makes it possible to know the degree of digital maturity of a territory, be it the whole of Catalonia, a province or a county.
• Get an overview (who is leading the way and who needs help) and understand the context of public sector digitization (strengths and areas for improvement).
• Others: create a data-driven culture, identify best practices, value the work done.

In addition, it is particularly useful for organizations that must provide support in the digital transformation of local entities (such as the AOC!), as it allows the identification of the objectives on which to act and facilitates the design and analysis of the effectiveness of digitization policies.

Until 2018, the AOC calculated the degree of digitization of local bodies from the  indicators of the Electronic Administration Map. These indicators were obtained from the implementation data of the AOC services and the analysis of the municipal websites. All were "attribute" type indicators, that is to say, they indicate qualities or states that the entity analyzed may or may not fulfill (e.g. it has a citizen folder, e-mail address, contractor profile...; it offers online procedures ) . Over time we realized that the adoption of digital services by local bodies was becoming more widespread and that assessing the degree of implementation was not enough. It was necessary to look for new indicators that reflected the progress in the digital transformation, such as the use they made of these digital services.

In 2018, activity indicators were introduced, i.e. “variables” that can be quantified, such as:

• Inquiries at Via Oberta
• EACAT Procedures (or SIR Procedures)
• e-TRAM procedures (including the generic transfer and the FUE channel)
• e-FACT invoices
• e-NOTUM notifications
• Use of T-CATs (Validator)
• Publications in the e-TAULER
• Publications on the Contractor's profile (electronic tenders)
• Publications/updates to the transparency portal
• Submissions from the IVTM and/or TESTRA

However, in a context as changing as the current one, the way to evaluate electronic government must continuously evolve to remain effective and incorporate new aspects that respond to the reality of the moment, based on empirical evidence. The e-Government reports and maps project thus entered a new phase at the end of 2019 and was replaced by an index that we named theDigital Maturity Index.

Thanks to the implementation of business analytics (Business Analytics) in the management of AOC services, it was possible to automate the collection and integration of the activity indicators of the index and to present the information through an interactive visualization of the data. In addition, in the 2020 edition a significant improvement was made in the evaluation of the use of digital services: instead of assessing the use of a digital service based on the absolute number of transactions, we began to measure the digitization associated with the use of the service; that is, the calculation as the number of digital transactions relative to the total number of transactions, including face-to-face transactions (for example, electronic entries relative to total entries, or electronically bid contracts relative to total bid contracts).

The IMD, a collaborative tool in constant evolution

In order for the IMD to be useful and informative, it must always be borne in mind that only the most relevant indicators of the moment in which we find ourselves should be included, those that mark the difference between an entity that we consider digitally mature and a other than it is not. And the weights of each attribute or variable within the index must be determined very carefully. To provide the IMD with a rigorous and flexible calculation methodology (able to adapt to the present moment), a working group of experts in public management was created and the result was shared and debated with the deputations , county councils and town halls.

In addition, activities and meetings of staff from the local world are periodically organized to review and evolve the IMD, such as the workshop that took place at the Digital Government Congress 2021, in which more than 50 representatives of local Catalan administrations participated and where improvement proposals were collected, some of which we have already incorporated into the 2023 IMD, or the workshop held in November 2023 with representatives of town councils and councils, to revise and improve the IMD.

Therefore, the IMD is the result of collaborative work, always applying a continuous improvement approach with feedback from public staff and continuous investment in creating a datawarehouse that integrates data from different sources.

IMD calculation methodology 

The IMD is a composite index or indicator consisting of more than 30 indicators organized into three dimensions and nine sub-dimensions that are aggregated in a weighted manner to obtain a comparable value. It is currently calculated for 947 councils and 41 county councils.

The IMD is conceptually divided into three dimensions:

— Digital rights. We assess the availability of digital public services that allow citizens to exercise their digital rights and compliance with the regulations governing these rights.

— Digital services activity. We evaluate the use of digital services by citizens, companies, associations and public employees. Specifically, we measure digitization in the relationship with users, the digitization of internal processes and interoperability between public administrations.

— Open government. We assess the practical application of the principles of open government: transparency, access rights, open data and citizen participation.

The following figure schematically shows the three dimensions of the IMD, as well as their sub-dimensions and the indicators that make them up:

The IMD is the weighted sum of the values ​​of the indicators in each of the three dimensions:

Each indicator is assigned a weight and a maximum score based on its importance. The different weights have been assigned to 1) recognize (or penalize) the dimensions and sub-dimensions that contribute more (or less) to the digital transformation, according to people's opinion; and 2) reflect the statistical quality of the data.

See APPENDIX Data sources and types of indicators.

Presentation of the results

The IMD is presented through a set of dashboards and reports intended for the different profiles involved in the decision-making of a local body.

• The dashboard "We make a digital country" shows the key indicators of a specific territory in aggregate form (all of Catalonia, a province or a county). In addition to the IMD's own indicators, it shows others that derive from it, such as the percentage of entities that process electronically or that have a transparency portal.

• The dashboard "Your entity - Global vision" provides a quick and clear view of the state of each entity with the aim of enabling municipal elected officials to make data-based decisions. In addition to the total score of the IMD and its dimensions, it shows the value of key indicators and compares them with the average value of local entities in the same population band.

• The report "Your Ens – Detailed View" provides the complete picture of the IMD. It shows the value of all the indicators of a local body and also places them in the context of the municipalities in its population band. It is intended for managerial staff and municipal technical staff so that they can identify the strong points and areas of improvement on which to act.

• The "General comparison" report offers a global view of the degree of digital maturity of a territory. It allows supra-municipal bodies to identify where digital transformation policies need to be deployed for each of the three main dimensions of the index.

Acknowledgments Open Administration

We use the score obtained in the IMD to award every year the Open Administration Awards to the leading administrations in digital transformation, divided into seven categories according to population rank.

More information about the Acknowledgments Open Administration.

conclusions

The sustainability of this initiative is based on promoting that all the information required to produce the Index is available in open data formats based on standards, so that they can be easily obtained and integrated. Currently, 95% of the required information meets these requirements.

The IMD is a collaborative initiative involving different levels of government. The four Provincial Councils of Catalonia and all the Catalan regional councils participate in the validation of the MD's methodology and results. Good governance is essential in this project. Regarding the usefulness of the IMD, we have verified that the publication of the results, and especially the annual recognitions, serve as a stimulus for local administrations to identify their areas of improvement and advance in the digital transformation process.

Status of the project (in production)

Every year, since 2019, the AOC analyzes the digital maturity of the local public body applying the IMD methodology. The result can be consulted on the website of theDigital Maturity Index.

More information

• Article "Digital maturity index of local administrations: a key tool for evaluation and continuous improvement” published in the Basque Journal of Management of People and Public Organizations of VAT (December 2022)
• Open Administration Recognition Methodology: https://www.aoc.cat/reconeixements-administracio-oberta-metodologia/
• E-administration reports (from 2010 to 2018):  https://www.aoc.cat/indicadors/informes-d-eadministracio-a-catalunya/

APPENDIX Data sources and types of indicators

Data sources

The IMD is calculated from public, objective and verifiable data obtained from the following information sources:

• The web pages of the local administrations themselves.
• The activity data of the digital services of the AOC available at Open Data Portal of the AOC.
• The activity data of the AGE services available on the Data OBSAE portal of the Electronic Administration Observatory (General Secretariat of Digital Administration).
• The activity data of digital services provided by the local entities themselves and by the providers of market platforms, which are subjected to a review process to contrast their quality.
• Data from the Infoparticipa Index of the Autonomous University of Catalonia and the Cybersecurity Exposure Index of the Cybersecurity Agency of Catalonia.
• The data published in the official information and documentation search engine of the Diputació de Barcelona (CIDO).
• The recruitment data provided by the Catalan Recruitment Consultative Board.
• Citizen participation data provided by providers of public electronic participation platforms (Diputacions de Barcelona and Girona) and market (supplier companies).

In the case of missing data, intermediate variables are used (for example, if we do not have the total number of registration entries of a City Council, we obtain the data from the number of inhabitants and a correlation coefficient) . All missing data imputation methods used have been validated based on their statistical properties.

IMD indicators

Below is a breakdown of what is analyzed and how each indicator is scored:

Digital rights

• Citizen folder:
  • If it is available to us from the citizen folder
  • 2,44 points if we have it and 0 points otherwise
• Municipal search engine:
  • If it is available to us as a search engine on your municipal website
  • 1,63 points if we have it and 0 points otherwise
• Online instance:
  • If the us has a generic instance to do procedures online
  • 4,07 points if he offers it to us and 0 points otherwise
• Online register:
  • If you allow us to carry out interoperability queries on your register
  • 4,07 points if he offers it to us and 0 points otherwise
• Map viewer:
  • If you have a cartographic viewer on the municipal website
  • 0,81 points if we have it and 0 points otherwise
• Urban viewer:
  • If it is available to us from the town planning viewer on the municipal website
  • 0,81 points if we have it and 0 points otherwise
• ER idCAT:
  • If we are an idCAT registration entity
  • 2,44 points if the agency is a registration entity and has issued idCAT certificates during the index calculation year, and 0 points otherwise
• WAS:
  • If you have a Business One-Stop Shop
  • 1,63 points if it is offered by the maximum number of procedures and the proportional score otherwise
• MUX:
  • If the system is integrated with the MUX Register service of the AOC
  • 2,44 points if the system is integrated with MUX Registry and 0 points otherwise
• Hub citizen folder:
  • If it is integrated with El meu espai
  • 2,44 points if your file manager has integrated it with El meu espai, 0,81 if it has integrated AOC services and 0 points if it has no integration
• Mailbox Accessibility:
  • If we have an electronic processing mailbox for accessibility issues
  • 0,81 points if we have it and 0 points otherwise
• Accessibility Seal:
  • If an accessibility statement has been published on the municipal website
  • 0,81 points if it has been published on the municipal website and 0 points otherwise
• ENI:
  • If a certificate of compliance from the ENI has been published on the municipal website
  • 2,44 points if an ENI compliance certificate has been issued to us and 0 points otherwise
• TO US:
  • If the electronic administration platform has a certificate of compliance with the ENS on the municipal website or in the National Cryptographic Center
  • 2,44 points if the ENS has published a certificate of compliance and 0 points otherwise
• Regulatory ordinance:
  • If it has been approved by an electronic administration ordinance
  • 1,63 points if it has been approved by us and 0 points otherwise
• Cyber ​​Security Vulnerability:
  • Score of the Vulnerability Level calculated by the Cybersecurity Agency of Catalonia.
  • 2,44 points if the indicator is 1 (zero exposure level) and the proportional score otherwise (the ratio between the exposure level calculated by the ACC and the indicator value in the IMD is the following: null =1, low = 0,75, medium = 0,5, high = 0,25 and critical = 0)
• Data Protection Officer:
  • If it is available to us from the data protection delegate and has communicated it to the APDCat.
  • 0,81 if the company has a data protection delegate and has communicated it to the APDCat and 0 otherwise
• It represents:
  • If it is available to us from the power of attorney register
  • 0,81 if the company has a power of attorney register and 0 otherwise

Service activity

• Input digitization:
  • Percentage of electronic entries with respect to the total number of entries (e-TRAM online procedures (or own system) + e-FACT e-invoices + EACAT remittances received / total entries)
  • 8,10 points if the percentage is 80% and the proportional score otherwise
• Output digitization:
  • Percentage of electronic departures with respect to the total number of departures (Notifications and e-NOTUM + EACAT transmissions sent / total departures)
  • 5,40 points if the percentage is 100% and the proportional score otherwise
• Internal e-signatures:
  • Ratio of public worker signatures with respect to the number of entries
  • 6,75 points if the ratio is in the highest percentile of the entity's population band and the proportional score otherwise (10 percentiles are established for each population band)
• Inquiries via Open:
  • Ratio of consultations carried out in Via Oberta per inhabitant
  • 12,60 points if the consultation rate of Via Oberta is in the highest percentile of the entity's population band and the proportional score otherwise (10 percentiles are established for each population band)
• Via Open Services:
  • Percentage of active Via Oberta services compared to the mandatory
  • 5,40 points if the percentage is 100% and the proportional score otherwise.
• E-tenders:
  • Percentage of electronic tenders for open contracts with respect to the total
  • 4,05 points if the percentage is 100% and the proportional score otherwise
• Publications on the Contractor's Profile:
  • Rate of publications on the contractor's profile per inhabitant
  • 2,03 points if the ratio is in the highest percentile of the entity's population band and the proportional score otherwise (3 percentiles are established for each population band)
• Publications in the Register of Contracts:
  • Rate of publications in the public register of contracts per inhabitant
  • 0,68 points if the ratio is in the highest percentile of the entity's population band and the proportional score otherwise (3 percentiles are established for each population band)

Open Government

• Compliance with transparency:
  • Score received in Infoparticipa
  • 8,13 points if infoparticipa score is 100% and proportional score otherwise
• Open Data:
  • If you publish your own open data to us
  • 1,63 points if we publish it and 0 points otherwise
• Citizen participation:
  • Number of citizen participation processes by electronic means carried out by the entity
  • 1,63 points if more than 7 participatory processes have been carried out, 0,98 if between 4 and 6 processes have been carried out, 0,49 if between 1 and 3 processes have been carried out and 0 if none have been carried out
• Electronic vote:
  • If electronic voting has been used in any citizen participation process
  • 0,49 points if it has been used by us and 0 points otherwise
• Transparency Portal:
  • If you have a transparency portal
  • 3,25 points if it is available to us and 0 otherwise
• Electronic board:
  • If you have an electronic bulletin board
  • 0,81 points if we have it and 0 otherwise
• Alert channel:
  • If we have an internal alert channel.
  • 2,44 points if we have it and 0 otherwise
• Access to public information:
  • If we have a specific electronic procedure for requests for access to public information
  • 1,63 if the institution has a specific electronic procedure for requests for access to public information.