A Taste of Chorus’s Secret Sauce: How Our System Teaches Itself

August 17, 2017 Micha Breakstone

One of the biggest challenges when creating novel Artificial Intelligence (AI) is building the technology so that it can adapt to a wide range of ever-evolving real-world problems. Over the past two years, Chorus.ai has developed a unique platform that transcribes and measures what the best sales reps are doing so that it can surface winning sales patterns for managers to replicate across teams, and alert executives in real-time when deals are at risk.

To achieve this, i.e., decipher the DNA of winning sales patterns, we could not have relied on a system that is entirely predefined (either rule-based or Machine Learning-based). There are two main reasons for this. First, the sheer amount of nuance that the system needs to learn is staggering and prohibitive for rule-based systems. There are billions of ways to ask questions, raise objections, set action items, challenge hypotheses, etc. all of which need to be identified if sales patterns are to be codified. Second, signals and patterns evolve: new competitors, product names and features, and industry-related terminology change over time, and machine-learned models quickly become obsolete.

1 - Self Learning.png

The natural solution is to build a system that can teach itself, i.e. a system that contains an inherent feedback loop which can take one of two forms. The first form of a feedback loop is to have select high-confidence results of one component become part of the training set of a second component (see illustration to the right). The second is to have output of unsupervised aspects of our technology stack (i.e. those created without human tagging) modify the training set of supervised components (aka Transfer Learning). A system that does this can continue evolving and improving with little or no human intervention (assuming heuristics are employed to avoid cumulative error). Following are 3 examples of how we’ve implemented self-learning loops in our system.

2 - Learning Curve.png

Identifying Action Items

The Task: Identifying and extracting action items from sales conversations (e.g. “I’ll set time for us next week”, “Let me shoot over an email with the specs”, etc.)

The Approach:

Stage 1: We began by feeding our system with hundreds of manually selected examples of action items from which we learned a model that represents them with extremely high precision.

Stage 2: Having obtained high precision the challenge was to expand and capture new forms of action items that the system had not trained on (e.g. “I could definitely have that analysis prepared for you”). We did this by looking at various signals including patterns of words before and after the candidate phrases, as well as time-stamped notes salespeople took on the calls using the Chorus platform, and analyzing them for clues of new forms of action items.

Stage 3: After a few such semi-manual iterations, the system is mature and ready to go into the wild. By now it’s already quite robust and able to pick up thousands of forms of action items with great precision. New examples suggested by the notes on the system can now be added automatically with very high confidence. Effectively the system is now ready to teach itself.

Important Note: While the above describes the task of identifying “Action Items”, it can be applied in a similar manner to many other tasks such as pricing, discounting, delivering the pitch, and other key moments.

Automatically Customizing Speech Recognition Engines

The Task: Create a Speech Recognition engine with superior accuracy

The Approach: We employ self-learning for speech in 2 ways:

  1. For every utterance (spoken sentence) automatically transcribed we estimate our confidence in the transcription. We then collect millions of 99%+ high-confidence transcriptions and using Deep Learning (specifically Recurrent Neural Networks or RNNs) we model the language used by salespeople and feed that model back into the system to improve results.

  2. We have our system comb through related websites, extract words and phrases that are not in our engine’s vocabulary, and make learned guesses on how such words should be pronounced, adding them into the vocabulary and leading to even higher accuracy.

Results: We’ve seen tremendous success with this approach, improving upon state of the art engines by circa 20% in accuracy. For details see this post on Automatic Speech Recognition.

Identifying Winning Patterns

The Task: Uncover conversational patterns that have a greater chance of leading to a win

The Approach: Having identified hundreds of data points using approaches similar to that described for Action Items, we let the system analyze hundreds of thousands of conversations tagged for won/advanced vs. lost/didn’t-advance. We cross-categorize for stage, deal size, vertical, etc. to generate a hypothesis of winning patterns. As more data is collected we continually let the models evaluate and modify themselves to yield a better prediction of the actual results.

To summarize, Chorus.ai is building at the forefront of AI to unleash the power of Conversation Intelligence. To do this we are employing self-learning systems so that our data becomes our biggest asset as we drive our algorithms toward ever-improving accuracy.

Previous Article
“There’s no I in Closer” - How Top Performing Sales Reps use Language to close More Deals
“There’s no I in Closer” - How Top Performing Sales Reps use Language to close More Deals

Next Article
5 Things You Didn’t Know about Your Competition: Actionable Insights for Competitive Deals
5 Things You Didn’t Know about Your Competition: Actionable Insights for Competitive Deals