Chatlayer Documentation
Get in touchAbout Chatlayer
  • Welcome
  • What's new
  • Send feedback
  • start quickly
    • Leadzy tutorial
      • 0. Introduction
      • 1. New bot, new block
      • 2. Understand your users
      • 3. Collect and display user input
      • 4. Steer the conversation with Conditions
      • 5. Empower your bot with Actions
      • 6. Set up a channel and publish your bot
    • Templates
      • Banking
      • E-Commerce Returns
      • E-Bike Shop
      • E-Scooter Support
      • Feedback
      • Find Nearest Location
      • GPT
      • 🆕Gym
      • Job applications
      • Knowledge base AI template
      • Lead generation
      • 🆕NPS
      • Restaurant
      • Retail
  • navigation
    • Analytics
      • Dashboard
      • Customers
      • Conversations
      • User flow
      • Intents
      • 🆕Funnels [Beta]
    • Bot builder
      • Flows
        • Canvas functionalities
        • Collaborate with team members
        • Manage your flows
        • 🆕[Beta] Export and import flows
      • Bot dialogs view
      • Translations
      • Events
    • NLP
      • Dashboard
      • Intents
      • Expressions
      • Entities
      • Train
      • NLP settings
      • Improve
    • Knowledge base AI
      • Add content to your KBAI
      • Build your KBAI flow
      • Use your KBAI source URL in a button
      • Use tags to limit your KBAI content
      • 🆕Use Tables to store your KBAI questions
      • 🆕Manage handover where KBAI is unsatisfactory
    • History
      • Conversations
      • Execution logs
      • Versions
    • Channels
    • Tables
      • Create a table with records
      • Column types
      • Operate on your records
        • Insert record
        • Update record
        • Retrieve record
        • Delete record
      • API
    • Settings
      • Bot settings
      • Offloading
      • Variables
      • Voice
      • Import/Export
  • build your bot
    • Conversation design
      • Plan your bot
      • Conversation design workflow
    • Flow logic
      • Blocks
        • Message
        • Condition
        • Collect input
        • Action
      • Go-to connections
    • 🆕Emulator
    • Tips & best practices
      • Route your flow depending on the time
      • Improve the Not understood block
        • How to create better not understood messages
        • Not understood counter
        • Not understood Google search
      • Go to previous block
      • Recognizing a returning user
      • Skip introduction message
      • Route your flow depending on the channel
      • Reuse flows
      • Connect two bots to each other
  • understand users
    • Natural language processing (NLP)
      • Basic NLP concepts
      • Detect information with entities
        • Match entities
        • Contextual entities
        • System entities
        • Composite entities
      • NLP best practices
        • Word order and diversity of expressions
      • AI intent booster
      • Train your NLP
      • Context
      • NLP import & export
      • Sentiment analysis
    • Languages
      • Supported languages
      • Make your bot multilingual
      • Change language within the conversation
  • set up channels
    • Add a channel to your bot
    • All channels
      • 🚨Facebook Messenger [Deprecated soon]
        • Facebook Admin Removal
        • Facebook Webview Whitelisting
        • Facebook Messenger API updates for Europe
      • 🚨Google Assistant [Deprecated soon]
      • Webhook
      • Web
        • 🚨Web V1 [Deprecated]
          • Customize web widget
          • Live example web widget
        • 🆕Web V2
          • 🆕From Web V1 to V2
          • 🆕Web V2 methods and options
    • Sinch Conversation API
      • Configure your Sinch Conversation API
      • 🆕 Make the most of RCS with Carousels
      • WhatsApp Business API
  • integrate & code
    • API calls
      • Configure your API integration
      • Advanced API features
      • V1 API Reference
      • Custom integrations 101
    • Code editor
      • [Example] Sending data to Airtable (POST)
      • [Example] Retrieving data from Airtable (GET)
      • [Example] Making SOAP requests
    • App integrations
      • Airtable
      • Freshdesk
      • Freshdesk Messaging
      • 🆕Gmail
      • Intercom
      • Mailgun
      • OpenAI: GPT-3, ChatGPT and GPT-4
      • Salesforce
      • 🆕Slack
      • Topdesk
      • Zendesk Support
      • Zendesk Sunshine (beta)
    • 🆕Expression syntax
    • Human handover & live chat
      • Sinch Contact Pro
      • #Interact
      • Genesys Cloud
      • Offloading Webhook
      • Salesforce Service Cloud
      • Sparkcentral by Hootsuite
      • Zendesk Chat
    • Chatlayer messages specifications
  • Publish
    • Publish your bot
      • Publish a flow
  • Analyze & maintain
    • Analytics definitions
    • Data retention
    • Maintenance
    • Session
    • Track events for analytics
  • VOICE
    • 🆕Send bot response as audio
    • Voicebots
      • Create a voicebot
      • Voicebot-specific actions
      • Test your voicebot
      • From chat to voice
  • access & support
    • Identity & Access
      • Single Sign-On (SAML SSO)
    • Billing & subscription
    • Get in touch
    • Chatlayer glossary
    • SaaS Regions & IP Ranges
    • Status
    • Debug your bot
      • No correct response
      • Collect input not working
      • Video isn't working
Powered by GitBook
On this page
  • Word order
  • Word diversity
  • How to create optimal sets of expressions

Was this helpful?

  1. understand users
  2. Natural language processing (NLP)
  3. NLP best practices

Word order and diversity of expressions

This tutorial dives deeper on how to create optimal sets of expressions for your NLP model using word order and diversity of words.

PreviousNLP best practicesNextAI intent booster

Last updated 9 months ago

Was this helpful?

To ensure your chatbot effectively understands users, it must analyze two key elements:

  • the specific words used

  • their order in which these words are used

Word order

Word order can significantly impact meaning. For instance, I want to take the train, not the car differs from I want to take the car, not the train, despite containing the same words in a different order.

When generating expressions for your chatbot, it's crucial to account for all plausible word orders and syntactic structures.

Example

For instance, if you're feeding your book_train_ticket intent with expressions, make sure to account for different :

  • I want to take the train to Paris.

  • To Paris, I want to take the train.

  • I want to go to Paris by train.

While some structures might not be perfectly grammatical, it's important to include them, as users might not always be syntactically correct. If the meaning is clear to a human, it should be part of your model.

Word diversity

To ensure your chatbot understands the difference between good and bad inputs, use diverse words pertinent to the use case. For restricted cases, strictly control the vocabulary.

Word vectors

To make words usable by machine learning algorithms, they are converted into numerical vectors with hundreds or thousands of dimensions, similar to coordinates on a graph.

The advantage of this approach is that it naturally clusters words with similar meanings, such as "boat" and "ship," close to each other in the vector space. With the right techniques, word vectors can capture even more intricate relationships. For instance, "cars" and "planes" are both forms of transportation and will be related, yet their differences are also preserved.

If we put these modes of transportation on a simple graph, it looks something like this:

Example

For example, to design a chatbot for arranging land transportation, you must be precise with the horizontal axis—no water means no boat transport. The vertical axis can be more flexible; for instance, a bike suffices for small packages in crowded cities, whereas larger freight requires trucks or trains. The varying transport methods you would consider are highlighted in yellow below.

Consider another scenario where you need to transport something large, regardless of the mode of transport—land, water, or air. Here, cars and bikes are unsuitable, but a boat, train, or plane would be appropriate. In this scenario, you should allow some variety horizontally but limit vertical diversity, as depicted by the blue box in the illustration.

How to create optimal sets of expressions

Now that we know how important word diversity and word order are, we can use this insight to create a step-by-step guide on how to make a good set of expressions.

  1. Create a list of expressions with very different order of words. Try to come up with as many weird sentences as you can that might convey your intention of taking the train. Don’t worry if you forget something, you can always come back to this step later and add other sentences.

Example

If we look at the expressions shown above for taking the train, we can already use the sentences above and add a few more:

  1. I want to take the train to Paris

  2. To Paris i want to take the train

  3. I want to go to Paris by train

  4. By train is how I want to go to Paris

  5. To Paris by train is how I want to go

  6. …

  1. Be diverse. In this step, you take all of the sentences listed above and come up with as many synonyms as possible.

  • Start by chunking a general expression that represents your intent into a series of components.

Example

To illustrate how this works, let’s use the sentence: I want to go to Paris by train. First you strip this sentence of its components, like so:

  • I want to

  • go

  • to

  • Paris

  • by

  • train

  • Find synonyms for each of these componets. Here it's important to cover the highest word diversity possible for each word.

Example

Take I want to for example. You can replace this with:

  • I would like to

  • I must

  • I should

  • I prefer to

  • Let me

  • …

  1. Repeat this for every word or set of words.

Example
  • I want to go to Paris

  • I would like to travel in the direction of Paris

  • I should go to Paris

If later you come up with an additional synonym, you can just add it to the list of alternatives. That way you don’t have to check all your expressions again to see if you missed anything.

Following this approach you can quickly generate a large amount of expressions. If you look at the example above for “I want to go to Paris by train”, all possible combinations for this sentence amounts up to 6 x 3 x 3 x 2 x 2 x 3 = 648 expressions! So if you repeat this for all of the 5 sentences above, you'll end up with more than 3.000 expressions and you only had to come up with 5 or 6 sentences. Easy does it.

Word order and word diversity aren't the only elements to take into account when making optimal sets of expressions. Have a read at our list of as well.

To make this process easier, from your Intents tab..

To be optimal in your :

Using , this will look something like this:

Word order and word diversity aren't the only elements to take into account when making optimal sets of expressions. Have a read at our list of as well.

best practices
generate expressions
manual entry of expressions
expression generation
best practices
Example of how words are represented for machine learning.