In chatbots we trust—but should we?

Source: Supplied. Anna Collard, senior vice president, content strategist and ambassador of KnowBe4 Africa.

This remarkable progress has opened up a world of possibilities, but it also raises concerns about the reliability and accountability of these systems, warns Anna Collard, senior vice president, content strategist and ambassador at KnowBe4 Africa.

“While most users have reported positive experiences with chatbots, instances of factual inaccuracies, hallucinations, and potential privacy risks have surfaced,” says Collard. A recent study by Consumers International found that although 64% of participants would use chatbots again, a significant portion encountered reliability issues.

The cautionary tale of sports journalist Karien Jonckheere serves as a poignant example. Seeking inspiration for a cricket promo, Karien turned to ChatGPT, only to be met with a mix of accurate and fabricated match details that left her astounded. “It listed six matches, of which three were correct,” she remembers. “They completely fabricated the other three matches. It had given dates, venues, names of players, who scored the winning runs—all very specific, but completely made up.

“This is one of the obvious dangers of using chatbots,” comments Collard. “While they excel at answering straightforward questions, they can sometimes generate responses that are factually incorrect or nonsensical.”

Privacy and security are also crucial factors to consider. As chatbots become privy to our personal information and preferences, concerns arise regarding the protection and potential misuse of this data. “The stakes are high, as chatbots are no longer mere novelties but integral components of our digital ecosystem. From banking transactions to critical decision-making processes, their influence continues to grow,” says Collard.

Benefits of using chatbots

The advantages of chatbots for individuals and businesses are undeniable. “Chatbots are available around the clock, providing immediate responses to queries, which is especially useful for different time zones and those needing after-hours help,” explains Collard.

Their knack for providing prompt solutions to basic queries enables companies to leverage them on a large scale, resulting in significant cost savings. “Chatbots excel in managing repetitive tasks tirelessly, such as addressing common questions or assisting users through standard procedures,” remarks Collard. “They can juggle many interactions concurrently, a feat unattainable for a team of human agents.”

Risks of using chatbots

However, challenges arise when chatbots encounter complex inquiries beyond their scope. “It’s important to remember that they don’t understand what they’re saying,” explains Collard. “They collate information from all over the web and stitch it together to formulate a response.”

As Jonckheere soon realised, chatbots are—for now at least—ill-equipped at intricate problem-solving and can invent details when they are unsure of facts. “They also lack human intuition, leading to difficulties in grasping subtleties, sarcasm, and context,” Collard says. That is why chatbots could be programmed to transfer complex queries to human operators where relevant, Collard suggests.

“A robust chatbot system should offer the option to escalate the conversation to a human customer service representative for complex or sensitive queries that automated responses may not effectively address.”

Privacy and security considerations

Regarding privacy, it’s important to exercise caution when sharing sensitive details with a chatbot. That is why companies using these novel technologies should also have strict privacy compliance regulations in place.

“Chatbots typically collect user data to personalise interactions and improve services. And you may not want all your personal data being reused by the algorithm for other queries. Remember whatever we upload to public models such as ChatGPT, will be fed into their model, unless you specifically tick a setting that says otherwise.

Lerato Songelwa: Doubt - An unfortunate consequence of being a pioneer

1 day

For companies making use of chatbots, it’s crucial that it is managed like any other system, meaning restrict its access to information that it absolutely needs access to, and ensuring that personal information is stored securely and managed according to strict privacy regulations, such as Popia,” asserts Collard.

“For chatbots that handle sensitive transactions, such as banking queries, they should authenticate users before any personal information is accessed or shared.”

From a security perspective, it is important to test chatbots for prompt injection attacks before launching them to consumers. “Similar to other software, regular updates are essential for chatbots to address vulnerabilities that malicious actors could exploit,” notes Collard. “A purpose-built chatbot should also incorporate security measures to thwart automated threats such as spam bots.”

So, can we trust chatbots? “I like using chatbots, however when I use it for research or for anything where I need accurate data, I will always double check the original sources,” she expresses. “It is great that chatbots are available 24/7 and can help with a task or query at lightning speed.”

However, collaboration between humans and machines is key. “The crucial aspect is integrating chatbots in a manner that complements the strengths of human agents, enabling a seamless transition to live assistance when needed,” Collard concludes.

“Businesses can leverage the scalability and data-processing capabilities of chatbots, while users enjoy the convenience of instant and accessible service.”