In the midst of the rise of artificial intelligence, the implementation of a chatbot with generative AI can be key to improving the efficiency and satisfaction of citizens with public administrations. These chatbots represent an opportunity to optimize attention to citizens and businesses, resolve queries efficiently and offer a personalized and continuous experience.

However, to guarantee correct implementation, it is necessary to follow a series of good practices that ensure compatibility with existing systems, regulatory compliance and the quality of the responses.

This guide provides practical recommendations for implementing a chatbot with generative AI, based on our experience creating a chatbot with Generative AI. [1], ranging from selecting the right technology to its optimization and maintenance. It also addresses fundamental aspects such as security, data protection and continuous monitoring to ensure constant evolution of the system.

The objective of this guide is for public administrations to be able to deploy chatbots that not only respond to current needs, but can also evolve and adapt to future demands.

In this sense, we present this manual as a starting point for the creation of a chatbot with generative AI based on our experience. At the AOC we are also in a process of exploring this technology and this implies that there may be multiple aspects that can be improved, as well as different approaches to the development of a chatbot.

It is also important to keep in mind that this technology is constantly evolving, which means that new opportunities for improvement are constantly emerging, both in the short and long term. Therefore, it should be noted that this guide is a dynamic guide that will be updated as the AOC gains more knowledge and experience in this area.

To be able to explain how the current chatbot was implemented, it is especially important to look at the chatbots that, from the AOC, we used to support citizens.

During 2020, the AOC bet on the company 1millionbot [2] to make a first attempt at implementing traditional chatbots. In 2021, contract AOC-2021-99 was awarded: Service of a virtual conversational assistant (chatbot) [3] with the aim of providing a chatbot service to facilitate information consultation and processing processes related to AOC services.

Finally, during the months of May/June 2021, two virtual assistants for idCAT citizenship services, Certificat and VÀLid, were put into production, which reached an average of 50.000 annual users during 2022 and 2023.

Despite these numbers, these chatbots had certain shortcomings:

• The response intentions of these chatbots were very rigid and inflexible, not allowing for adaptability based on the user's response.
• Little capacity for adaptability to new problems faced by citizens, which ends up resulting in an excessive library of intentions.
• A chatbot was required for each service (in the services with the highest volume), which caused very costly maintenance.
• Costly training to derive user responses from one intent to another.
• Excessive hours spent training the chatbots due to the fact that the two bots had to be trained separately and each with its two corresponding languages ​​(Catalan and Spanish).
• The context of the conversations was not maintained, meaning that each interaction with the user was a new “conversation” for the bot.

These difficulties and shortcomings, along with the explosion of artificial intelligence in the last year, led the AOC to commission 1millionbot to begin working with an AI chatbot.

To ensure an effective implementation of a chatbot with generative AI, it is essential to select a solution that fits the organization's immediate objectives and that, at the same time, allows for long-term evolution and scalability.

In this case, the chatbot chosen by the AOC can manage a high volume of queries simultaneously and integrates easily with the services provided by the organization.

This chatbot is integrated into the AOC's corporate web pages using Google Tag Manager and is specifically deployed on support pages and services where it has the knowledge to provide answers to users. It is currently available at:

• IdCAT Certificate: https://idcat.cat/ i https://suport-idcat.aoc.cat/hc/ca
• Mobile IDCAT: https://idcatmobil.cat/ i https://suport-idcatmobil.aoc.cat/hc/ca
• e-NOTUM: https://usuari.enotum.cat/ i https://suport-enotum-ciutadania.aoc.cat/hc/ca
• It represents: https://representa.cat/ i https://suport-representa-ciutadania.aoc.cat/hc/ca
• e-FACT: https://efact.aoc.cat/bustia/home.htm i https://suport-efact-empreses.aoc.cat/hc/ca

Illustration 1 Location of the chatbot in the AOC services

3.1 Assessment of the organization's needs

Before selecting an artificial intelligence solution, it is essential to conduct a thorough assessment of the organization's needs.

This process should include the following key questions and aspects:

Chatbot objectives and needs

• • What are the main objectives that are sought to be achieved with the implementation of the chatbot? (Improve service to citizens, reduce the workload of teams, automate internal processes, etc.)
  • What current problems could be solved with this technology?
  • What impact is the chatbot expected to have on the organization's operations and strategy?

Target audience and use case

• Who are the main users of the chatbot? (citizens, companies, internal users of the organization, etc.).
• In which channels would this chatbot be integrated? (website, mobile application, messaging platforms such as Whatsapp or Telegram, etc.).
• What type of queries will the chatbot need to handle? (frequent and repetitive queries, specialized technical support, etc.).
• What budget can be allocated to this solution? It must be taken into account that a budget will have to be available for both its implementation and its maintenance.
• What are the possible limitations or risks associated with implementing the chatbot? Some of the risks to consider would be security and compliance with data protection.
• What are the available resources in the organization? It is important to ensure that the solution has sufficient human resources to not only implement it, but also to maintain it effectively. Training recommendations can be found in points 6 and 7 of this guide.

3.2 Scalability and flexibility

It is essential that the AI ​​solution is scalable and flexible to adapt to the changing needs of the organization. This implies the ability to handle an increase in the volume of queries at specific times of the year (period to apply for a grant or subsidy for example) without compromising the performance of the system.

It is also important that the solution allows for the addition of new functionalities and improvements as the organization grows and evolves. That is why it is essential to have not only the necessary internal resources, but also specialist maintenance support for the bot itself.

3.3 Compliance with security and data protection regulations

Security and data protection are critical aspects when implementing a chatbot with generative AI. The chosen solution must comply with current regulations on information security and data protection, such as the European Union's General Data Protection Regulation (GDPR). This includes:

• Guarantee the confidentiality, integrity and availability of data
• Implement appropriate security measures to prevent unauthorized access and cyberattacks
• Anonymization of personal data
• Establishment of a conversation custody period to improve the service.

Once the AI ​​solution has been selected, it is essential to assess whether it meets the technical and operational requirements necessary for its implementation and subsequent scalability. This system must guarantee continuous operation, offering 24/7 support and maintaining a high response rate.

4.1 Performance and capacity tests

Before final implementation, it is recommended to carry out:

• Performance and capacity testing to ensure that the chatbot can handle the expected volume of queries without problems.
• Load simulations to evaluate the behavior of the system under different traffic conditions in order to avoid possible bottlenecks or weak points.

4.2 Pilot tests

Pilot tests are essential to evaluate the performance of the chatbot in a controlled environment before its full implementation or when you want to implement a whole new service to offer.

These tests allow us to identify possible problems and adjust the system to improve its functionality.

In the case of the AOC, pilot tests may include:

• The simulation of different usage scenarios.
• The plot variation of the same incident.
• Analyzing the results to make necessary adjustments.

 

4.2 Secure and reliable infrastructure

To guarantee correct availability and security of the bot, the service infrastructure must be hosted on secure and reliable servers, within the European Economic Area, to comply with data protection and security regulations.

At the same time, if you want to have good security for this virtual agent, this must include the use of certified data centers and the implementation of regular backups to prevent data loss in the event of incidents.

5.1 Instructions

To ensure optimal performance, the chatbot must have a set of clear instructions that guide the AI ​​model in its interactions with users. These instructions should include specific guidelines on how to structure responses and how to prioritize relevant information.

As can be seen in the image, it is very important that the instructions serve to explain how to manage complex queries to avoid conflicts and confusion with citizens.

Illustration 3 AOC AI chatbot instructions

These instructions must be reviewed and updated periodically, not only to ensure that the chatbot adapts to the organization's new needs, but also to be able to correct errors and anticipate future problems.

5.1.1 Structure of the responses

The chatbot's responses must follow a clear and coherent structure to facilitate understanding by users. The chatbot must be able to:

• Use short and simple sentences.
• Divide the information into paragraphs or points to make it easier to read.
• Use accessible language.
• Avoid technicalities or complex terms that could generate confusion.
• Provide where the information came from so that citizens can consult the original source.

Illustration 4 Example of clear and coherent structure as well as sending the source of information

As you can see in the example, it is very important that, when the information is very extensive or simply seeks to expand the information, the chatbot always attaches the original FAQS as it provides added value and reliability in the response.

5.2 Personality

The chatbot must have a defined personality that adapts to the needs of the users and, above all, to the institutional image that the organization wants to give. Also, In accordance with point 27 of European Regulation (EU) 2024/1689 [4], these artificial intelligence systems must be developed and used in such a way that allows the people who use them to be aware that they are communicating or interacting with an AI.

It is essential to establish an appropriate tone of communication that is close, natural and aligned with users' expectations.

Illustration 5 AOC bot personality

This bot must maintain:

• Consistency in language and response style to offer a smooth and satisfying experience.
• Having not only the use of appropriate language, but also the ability to convey empathy in responses,
• Make users feel heard and valued.
• Offer to continue the conversation at all times in case the citizen needs additional information to the question asked.

Illustration 6 Example of information

 

Illustration 7 Offer to continue the conversation and provide more information

5.2.1 Definition of voice tone and adaptation of the chatbot

The bot's tone of voice should reflect the values ​​and culture of the organization. For example, a chatbot for a financial institution may opt for a more formal and professional tone, while one for a public administration may opt for a more approachable and conciliatory tone.

It is important that the tone of voice is consistent in all interactions to generate trust and credibility among users. This tone of voice must be able, not only to be stable and treat all people equally, but also to be able to adapt to different situations, always maintaining a positive and proactive attitude, even in moments of conflict or dissatisfaction on the part of the user.

The chatbot must be able to adapt to different types of users taking into account factors such as:

• Age
• The level of technological knowledge
• Personal preferences
• The type of service/product for which they need help

To achieve this adaptability, and ultimately a more satisfying and relevant user experience, it is essential to personalize responses through generative intentions.

Illustration 8 Adapting the bot to a person with little technological knowledge

5.3 Knowledge Library

The chatbot's knowledge should be based on a combination of reliable sources such as FAQs, documentary databases, up-to-date information systems, and previous user incidents. This knowledge library should be easily accessible and navigable to allow the chatbot to find the necessary information quickly and efficiently.

In the case of the AOC, the chatbot's knowledge library contains the following information:

• Links to all FAQs on the AOC Support Portal.
• Instructions for answering complex questions
• Real user incidents with Question/Answer format

The diversity of these information sources allows the bot to provide more complete and well-founded answers. Additionally, using sources also helps ensure that the chatbot can respond to a wide range of queries and provide up-to-date and accurate information at all times.

Illustration 9 User incidence with Question/Answer format

 

Illustration 10 Instructions for answering complex questions

It is essential that this knowledge library is kept constantly updated to ensure the accuracy and relevance of the answers. This involves:

• Periodically review information sources
• Incorporation of new content as it is generated or modified.
• Establish an information validation process to ensure that the data used is correct and reliable.

5.4 Creating Generative Intentions

It is crucial to precisely define the main intentions of the chatbot in order to ensure flexible and efficient interpretation of users' questions. Unlike traditional chatbots, a bot with generative AI must be able to understand queries formulated in different ways and adapt the response according to the context.

Creating a generative intent is not far from creating traditional intents, the main difference is that small instructions can be added to guide the bot into a more personalized response.

Illustration 12 Generative intent error accessing a notification

As seen in the image, first some training phrases are defined (just like in classic intentions) and then a series of instructions are written so that the bot generates a generative response. It should be noted that the AI ​​allows for variations of the entered training phrase to be generated without having to create them one by one.

This needs to be clearly and precisely defined to ensure correct interpretation of user queries. To do this, you need to identify the most frequently asked questions and common needs of users, and create specific intents for each of them.

Filters should be implemented to detect and manage queries outside the scope of the service, providing alternative recommendations to users when necessary. This may include redirecting to other support channels, suggesting additional resources, or referring to a human agent when necessary. In addition, the chatbot should be able to recognize when a query is too complex to be resolved automatically.

This intention of being out of the catalog is key so that the citizen is clear about the scope of the service and the knowledge of the chatbot. It is important not to generate false expectations and make it clear that the user must contact the corresponding administration, always offering them more information about the AOC services.

Illustration 13 Intent outside the scope of the AOC and suggestion for additional resource

It is important to review and update intentions regularly to adapt to new needs. It is important to keep in mind that the bot must be able to manage multiple intentions simultaneously in the same conversation, offering coherent and relevant responses at all times.

Finally, having well-structured and clear intentions will allow you to offer a more satisfactory user experience and avoid frustrations.

5.4.1 Adaptability and flexibility

The bot must be able to adapt to different ways in which users formulate queries. This implies the ability to recognize synonyms, grammatical variations, and different sentence structures.

It is also important that the chatbot can adapt its responses according to the context of the conversation, offering relevant and accurate information at all times. Likewise, it must be able to learn from previous interactions and adjust its responses based on the specific preferences and needs of each user.

Illustration 14 Examples of flexibility in variations and sentence structure

Illustration 15 Examples of flexibility in variations and sentence structure

5.4.2 Alternative channels of attention

To avoid exclusive dependence on an automated system, it is important to offer alternative channels of personalized attention.

This allows users to access human support in cases where the chatbot cannot provide an adequate response or when more detailed attention is required. Alternative channels can include phone support, email, or live chat with human agents.

Illustration 16 Offering alternative contact channels

6.1 Data protection, minimization and deletion

The chatbot must ensure data minimization, avoiding the collection of unnecessary personal information. All conversations must be anonymized and deleted after a set period of time to avoid data protection risks.

In the case of the AOC, conversations are deleted after 3 months since it is considered that sufficient time has passed to analyze the responses. At all times, users are transparently informed about data collection and improvement.

Illustration 17 Welcome message about data processing

Data minimization is a fundamental principle in the protection of personal data. The chatbot should collect only the information strictly necessary to provide the service, avoiding the collection of sensitive or unnecessary data. This helps reduce the risks associated with data protection and comply with current regulations.

To guarantee user privacy, all conversations must be anonymized, removing any information that could personally identify users.

For example, in the AOC at the citizen and business level, when the user provides some personal data, the chatbot tells them that they are entering sensitive data, understanding that this data is essential to carry out the procedure.

The AOC is working to have anonymization mechanisms in place when a user adds personal data. This is important because, as conversations are kept for 3 months, not storing the personal data entered by the user adds an extra layer of security.

 

Illustration 18 Personal data citizen view

6.2 Data protection, minimization and deletion

To comply with the European Union AI Regulation, the chatbot must be clearly identifiable as a virtual assistant and must provide transparent information about its operation.

Publishing an algorithmic transparency sheet allows users to know what data was used to train the system and which AI model is being used. It also ensures that the system does not make decisions that have a significant impact on users without human oversight.

6.2.2 Algorithmic transparency

To guarantee user trust in the AI ​​system, the AOC recommends and defends full technological transparency through our algorithmic transparency sheet. [5]It should be noted that this sheet is constantly evolving and updated whenever there is a relevant change in the chatbot.

This transparency is essential to ensure user trust in the AI ​​system. This involves providing detailed information about how the chatbot works, including the data used to train the model and the algorithms employed.

Publishing an algorithmic transparency sheet allows users to understand how decisions are made and what factors influence the chatbot's responses.

6.2.2 Human supervision

This involves establishing mechanisms for review and human intervention in cases where the chatbot's decisions could have significant consequences for users. Human oversight helps ensure that decisions are fair, ethical, and aligned with the organization's values.

To comply with the European Union AI Regulation, it is necessary to ensure that the chatbot does not make decisions with a significant impact on users without human supervision.

A chatbot with generative AI must be constantly evolving to improve the quality of its responses and adapt to new user needs. This requires a continuous monitoring process that includes:

• The conversation review
• Detecting error patterns
• The optimization of generative intentions.
• Identify areas for improvement and add new knowledge to the system database.

7.1 Review routines

7.1.1 Reviewing conversations

It is essential to conduct a periodic review of the conversations generated by the chatbot to identify incorrect responses, inconsistencies or problems with understanding. This review should include the analysis of random samples to ensure the overall quality of the service. These random samples should take into account the following factors:

• Day of the week
• Time of day
• Type of service
• Search conversations by keywords

The time dedicated to these reviews must depend on the resources available to the organization; in the case of the AOC, approximately 1 hour per day is dedicated to everything implied in point 6 on review routines.

It is very important to document detected errors and implemented corrective actions to continuously improve chatbot performance.

7.1.2 Analysis of error patterns

Error pattern analysis allows you to identify recurring problems in the chatbot's responses and develop specific solutions to correct them. To detect these error patterns, it is crucial, once the erroneous conversation is detected, to search for its keyword and thus identify other conversations with the same error.

By doing this, it will later be much easier to adapt the corresponding generative intention and incorporate new information into the knowledge library.

7.1.3 Optimization of generative intentions and incorporation of new knowledge

Once the error patterns have been detected, it is necessary to optimize the generative intentions by adjusting the instructions of each one with the aim of ensuring that it can effectively respond to a wide variety of queries.

For example, initially the idCAT Certificate renewal intention did not include links to further information. Once the error was detected, in the intention instructions you added “Always respond with: How do I renew the idCAT Certificate?” and once you ask the bot again it will always attach the information from the FAQ.

Illustration 19 Insert new information into the bot instructions

Illustration 20 Bot response with link to renewal FAQ

Incorporating new knowledge into the chatbot library is also essential to maintain its relevance and accuracy. For example, if a new FAQ is generated on one of the support portals, it is essential to update the existing information, incorporating this new source of knowledge.

7.2 Activity monitoring

7.2.1 Statistics collection

Monitoring chatbot activity is key to understanding its usage and detecting potential bottlenecks in user interaction. Collecting statistics on frequency of use, average conversation duration, percentage of successfully resolved questions, and most common queries allows you to make informed decisions to improve the system. This data can also help identify peak times and optimize the availability of technology resources.

Illustration 21 AOC chatbot statistics screen

These analyses also allow us to identify trends and patterns in the use of the bot, especially in the most frequent queries and answers and the areas where we have the most difficulty in providing appropriate answers.

7.3 User ratings

7.3.1 Satisfaction surveys and scoring systems

It is key that this bot or the organization implements mechanisms for evaluating user responses, such as satisfaction surveys or scoring systems. This allows:

• Obtain an objective view of the degree of user satisfaction
• Make adjustments to the bot's operation
• Rate chatbot responses quickly and easily

Satisfaction surveys may include questions about the quality of responses or include the use of stars, points, or other metrics to indicate the degree of satisfaction with the response received.

To ensure that the chatbot meets the organization's expectations, it is necessary to establish a constant evaluation system. This includes defining key performance indicators (KPIs) such as success rate, response time, and reduction of support tickets. It is also important to establish periodic comparisons with other support channels to assess whether the chatbot is providing added value.

Pilot tests can be carried out at different times of the year to identify possible improvements in its functionality and ensure that the user experience is always satisfactory.

To complement this process, the following must be created:

• A periodic evaluation methodology in which real user conversations are reviewed to detect recurring errors.
• Analyze the quality of the responses (point 6 of the guide).
• Implement adjustments that improve the overall performance of the chatbot (point 6 of the guide)

8.1 Reduction of support tickets and periodic comparisons with other support channels

It should be noted that the AI ​​bot is also a support mechanism that extends traditional communication channels. Having an AI bot will in no way completely replace human support, but rather adds a previous (close and friendly) layer to this human support.

Support ticket reduction is an indicator that measures the effectiveness of the chatbot in resolving queries without the need for human intervention. A significant reduction in the number of support tickets indicates that the chatbot is providing added value to the organization, improving operational efficiency and reducing the workload of human agents.

In the case of the AOC, although it is true that it cannot be said that it was exclusively due to the chatbot since other support improvements have been implemented, in 2024 support requests have been reduced from 93.000 in 2023 to 65.000 in 2024. Likewise, while support requests have decreased, transactions and the number of queries have doubled in the same period.[6].

Illustration 22 Evolution of support requests

Making periodic comparisons with other support channels such as telephone support, email or chat with an agent allows you to identify the strengths and weaknesses of the chatbot and determine if it is providing added value compared to other channels.

These comparisons also allow us to identify whether new incidents are occurring that were not anticipated by the bot in order to adjust the generative intentions or improve the knowledge library.

The goal is to ensure that the chatbot offers a satisfactory user experience and is aligned with the organization's expectations.

8.2 Periodic evaluation methodology

8.2.1 Review of real conversations

To ensure that the chatbot meets the organization's expectations, a periodic evaluation methodology must be established in which real user conversations are reviewed. Doing this allows you to detect recurring errors, analyze the quality of responses, and implement adjustments that improve the chatbot's overall performance.

It is important to document this methodology along with the way in which these conversations will be analyzed. In the case of the AOC, the methodology followed is as follows:

• Review period: Monthly
• Number of conversations to review: 150 conversations
• Review frequency: 5 random conversations for each of the 30 days of the month.
• Daily review time slots: 1 random conversation is reviewed within the morning, noon, afternoon, evening and night hours.
• Question to rate the answer: Did the bot correctly solve the citizen's problem? YES / NO
  • YES: When the bot has correctly answered everything the user has asked.
  • NO: When the bot has answered incorrectly or does not respond to what the user has requested.
• In the case of conversations with more than one interaction, the total count of the bot's responses will be assessed to conclude whether the bot has helped the citizen or not. If one of the interactions has been incorrect by the bot, it is assessed with a DO NOT the conversation.
• It is important to record interactions that have been outside the chatbot's competence to quantify how many queries are not treatable by the bot.
• You must indicate the service/product for which the chatbot has provided assistance.

These conversation reviews help identify new user needs and adapt the chatbot to meet them.

1. AOC Blog – The aoc implements a leading citizen attention chatbot using generative AI (https://www.aoc.cat/blog/2024/xatbot-aoc-iagenerativa/)
2. https://1millionbot.com/
3. Virtual conversational assistant service (chatbot)(Publication information – Public Procurement Services Platform)
4. Regulation (EU) 2024/1689 of the European Parliament and of the Council, of 13 June 2024, establishing harmonized rules in the field of artificial intelligence and amending Regulations (EC) nº 300/2008, (UE) nº 167/2013, (UE) nº 168/2013, (UE) 2018/858, (EU) 2018/1139 and (UE) 2019/2144 and Directives 2014/90/UE, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Regulation) https://www.boe.es/buscar/doc.php?id=DOUE-L-2024-81079
5. Algorithmic Transparency: AOC's Generative AI Chatbots (https://www.aoc.cat/ia-transparencia-xatbots-amb-ia-generativa/)
6. The innovative generative AI chatbot allows the AOC to double the number of requests handled by the support service (https://www.aoc.cat/blog/2025/xatbot-ia-generativa-suport/)