Innovation Days

Innovating Market Intelligence with AI and Data Analysis

Written by:

Jordi Vila

2023年10月18日

Motivation

At Fluendo, our dedication to Innovation Days and Market Intelligence reflects our commitment to providing pivotal market insights. We focus on obtaining and refining data to enable data-driven decision-making, improving efficiency, and fostering a customer-centric approach. By doing so, we aim to give Fluendo a competitive edge in the marketplace.

Proposal

To provide Fluendo with an edge, we are introducing a sophisticated system that blends Automated Data Gathering with Real-Time Market Analysis. By tapping into the vast expanses of the internet and aggregating data from myriad sources, we're positioned to draw actionable insights. Our intelligent agent is designed to distill this data, emphasizing relevant information and filtering out the superfluous. Complementing this, our real-time market analysis capability ensures that we're always attuned to shifts in consumer behavior, competitor strategies, and overarching market dynamics.

API development

We designed our API development with efficiency and adaptability in mind, and this image underscores our strategy. Using NVIDIA GPUs, we've ensured reliable performance without unnecessary complexity. Docker represents our practical approach to ensuring consistent operations, and with CUDA, we've streamlined processes for better speed. The variety of software tools, from natural language processing to machine learning and visual recognition, indicates our goal: to build versatile and effective digital solutions without the pomp and fanfare.

Evaluation

Our evaluation strategy is anchored in a rigorous testing framework. We initiated this by formulating a set of pertinent questions spanning six primary topics:

Customers (red)
Industries (orange)
Competitors (yellow)
About Fluendo (light green)
Events (dark green)
Other (blue)

Leveraging a confusion matrix, we gauged the accuracy of each response, focusing on metrics like precision and recall.

Recognizing the pivotal role of prompt engineering, our experiment was structured in three iterative loops. Our starting point, Q_0, consisted of questions without prompt engineering. We refined these questions to align more closely with our anticipated answers.

To do so, we used ChatGPT 3.5 itself and followed this strategy:

“I’m creating a list of prompts to automate some tasks. To evaluate the model performance, I defined the expected answers, but I’m not getting the expected answers. On the next questions, I will write you the question (q) and expected answer (a), and you should propose a prompt (p) to obtain the expected response. Do you understand?”

Here, you can see the results for both sets of questions using the ChatGPT 3.5 model:

The results using the ChatGPT 3.5 model were illuminating. A marked uptick in both precision and recall was observed, with metrics soaring from an initial average of around 40% to over 70%.

A subsequent loop of prompt engineering, labeled Q_2, introduced ChatGPT 4 and Bing into the mix. The first iteration of prompt engineering yielded significant improvements, but the second iteration's impact was more muted. Interestingly, the inclusion of internet access via the OpenAI API seemed to degrade performance, likely attributable to integration challenges with LangChain and SerAPI.

For this second and last set of engineered prompts, we added ChatGPT 4 and Bing (with native internet access).

Chart

As one can see, the first loop of prompt engineering does a very good job, while the second does not affect the results in any significant way.

Moreover, we can see a performance degradation when adding internet access to the OpenAI API. That is most probable due to poor implementation of LangChain and SerAPI and requires a deeper understanding from our side.

When focusing on the models, we see an improvement between ChatGPT 3.5 and 4, but not a significant one and certainly not one that justifies a 10x price increase. The best-performing model is Bing, which is free and runs ChatGPT 4. Its intrinsic internet access is a significant advantage, but its inaccessibility via API is a limitation to consider.

Next Steps

In order to take full advantage of this work, we still need to improve our understanding of Langchain and connect it to different APIs to increase accuracy and functionality like predictive analysis and data visualization. Afterward, we must implement a GUI, standardize ResearchRaptor as a search tool for the Mkt&Sales departments, and generate metrics to follow its performance.

Conclusions

Our exploration into innovation and market intelligence has reinforced the pressing need for advanced solutions in today's fast-paced environment. Drawing from our findings, we recommend creating a prototype of an AI-enabled market research tool. This will not only validate our hypothesis but also provide a tangible representation of our vision.

Our trials with Bing showed it has potential, but its lack of API access limits its potential. We also faced challenges with LangChain and SerAPI, so we have areas to improve. Overall, these experiences are helping us refine our approach and better serve our customers by harnessing the power of AI.

By crafting demonstrative use cases and real-world applications, we can highlight the tangible benefits of AI, from precision in data analysis to predictive market trends.