FAQ Chatbot on Facebook Messenger
With Flask，sklearn and MongoDB
This summer I was so honored to have a chance and work with Massachusetts Immigrant & Refugee Advocacy Coalition (MIRA) to continue their census survey and services which were dramatically affected by the COVID-19 pandemic. I led a team that is responsible to build chatbots on some popular social media platforms, including Facebook and WhatsApp, among the immigrant communities. And this article is a quick rundown of the process that I went through and the solutions to some questions that I met during the development.
Upshot: until I publish this article, the representative of my client MIRA has not granted access to this chatbot to all the facebook users. I will edit this article when it fully functions.
Data Collection
Our target is developing a chatbot that is expected to be smart enough to answer human users’ questions on its own. So obviously, we need the previous FAQs and their standard answers for training. Luckily, we had access to the data with help from the client, which saved us a huge bunch of time. What was left in this stage is translating them into two other languages: Chinese (traditional/simplified) and Spanish because our major target users were the immigrants whereas the Chinese and Spanish speakers make up a large part of them. As the only Chinese speaker in the team and a starter Spanish speaker, I took on part of this responsibility, which took nostalgic me back to the high school time and was a good chance to practice the language I was learning.
Data Cleaning
I utilized the TFIDF related methods of the powerful python library sklearn to implement this step.
Since we had two columns: questions and answers, I assumed there are three scenarios where the bot can respond: a) the question asked by the user is similar to a certain one in the dataset, then we just return the answer to it; b) the question and a certain answer…
