3. Detecting information in expressions

In the previous tutorial, you learned about intents and expressions. In this section, we'll cover how to gather user input by using entities, or categories of information.
📚 A bit of theory: entities
As we saw in the previous tutorial, our NLP engine is what makes bots able to understand human language. Depending on what are the bot use cases, the conversation will probably always refer to the same categories of things. 
Those categories of information are called entities in machine learning. An entity is any string that consistently refers to the same pre-defined category of objects. 
In the case of ChooChoo, the bot use case is to book a train ticket. This means that conversations will very likely be about which city to depart from, which city to arrive at, or even the date of travel. Therefore, entities in ChooChoo could be destination, origin, or date.
Example of an expressions where @date, @origin, and @destination entities are extracted.
The example above showcases tomorrow as an instance of the entity @date, Brussels as an instance for @origin, and Paris as an instance of @destination. To teach your NLP engine to recognize those as instances of entities you will need to provide many examples of instances for every entity. 
On Chatlayer, the convention to write the name of an entity is with an @ sign in front of it, e.g. @date.
Let's add entities to our ChooChoo bot 👇
Step 7: Creating contextual entities
Let's say we have an intent where the user wants to book a train ticket. A few different expressions could be: 
I want a train ticket
I need a ticket
Can I book a train ticket here?
If we feed the NLP model with these expressions above, then it will understand that the user wants to book a train ticket.
But what would happen if the user says:
I want a train ticket to Amsterdam
I need to go to Antwerp tomorrow
Can I book a train ticket to Brussels please?
The bot still understands the user's desire to book a train, yet these expressions contain extra valuable information. We want to make sure we capture that information ( in this case destination and time), to save them as contextual entities.
In this tutorial, we will use contextual entities, i.e. entities detected depending on the context of the sentence. But what does that mean? For example, city names in the context of a ticket booking flow may refer to a departure or destination. Yet in another context, for instance an e-commerce bot, a city can be a delivery address component.
Create book train ticket intent
Not all users will immediately mention their destination, so let's make sure we train our intent both with expressions that contain and do not contain entities:
Add a new intent called book train ticket
Add some simple expressions, like:
I want a train ticket
I need a ticket
Can I book a train ticket, please?
If you have trouble creating an intent and adding expressions, please read the previous tutorial.
Add entities to expressions
Now it's time to add contextual entities.
Go to Intents and select your book train ticket intent
Click on + Add expression to create a new expression
Enter an expression that contains an entity, for example: I want to book a train ticket from Brussels to Paris
Select Brussels in this sentence
Click on the '+ entity' icon in the bottom right of the expression box to create a new contextual entity for 'Brussels'  
The '+ entity' icon
Brussels is the location the user wants to depart from, so we will name this entity origin
Type origin in the Create new entity field and click on 'Create new entity' to confirm
Creating a new entity called 'origin'
Brussels will be added to the list of possible values for the @origin variable
Do the same thing for Paris as a 'destination' entity
You will then have the following set-up for this expression:
Creating multiple entities in an expression
We now save added the expression 'I want to book a ticket from @origin to @destination', where 'Brussels' is a value for @origin and 'Paris' is a value for the entity @destination. 
Add some other values to the 'origin' and 'destination' entities in the expression field. These will be saved for all future expressions. You can add these in the 'Create new value' box and pressing enter.
Add more expressions that contain the entities origin and destination
Once you have added more Entity values, these will also show up in the menu 
Entities > Contextual Entities
Now, let's add some more expressions to our Book train ticket intent. Some ideas for expressions:
Can I book a train from Cologne to Brussels?
I need to be in Rotterdam
I need a train to London
I want to travel to Lyon
I want to buy a ticket from Moscow to Vladivostok
I need a ticket from New York to Baltimore
When typing a new expression, you can create entities and add values to them in two ways:
1.Typing @ and then the name of the entity, for example @origin. You can add a new value in the box below with 'Create new value'
2.Selecting an entity value and clicking the +entity value button. For example, select 'Cologne', click the +button. This will result in 'Cologne' being changed into @origin and 'Cologne' will be a value of @origin
Make sure you retrain the NLP model by clicking the Update NLP button in the right upper corner.
This will now result some expressions for the Book train ticket intent, and entity values, like so:
Adding expressions to an intent
Note that Chatlayer offers 4 types of entities, you can read more about them here.
Step 8: Testing contextual entities
After we have retrained our model, let's see if its good enough to recognise the destination entity.
Go to NLP >Test to open the testing console 
Write 'Can I book a ticket from Antwerp to Paris?' as the expression to be tested
Click on Test
Testing an expression
You'll see that the entity @origin gets recognized with a 90.11% confidence score, and @destination with a 63.99% confidence score. 
The results in the Test page will be different based on your training set. If your entities are not recognized correctly, you can add some in the Entities window under the NLP tab. Make sure you retrain the NLP model before testing newly added expressions!
Now we know how to add intents, create expressions and entities, however we still need to create a conversation so that ChooChoo replies accordingly. Let's add some bot messages in the next step!
Step 9: Using variables in messages
When a user says something containing an entity, and the entity is successfully detected, our tool will automatically store the entity as a variable for that specific user.  
It is interesting to extract entities from the user expressions if they are useful pieces of information to handle in the conversation. To do so, entity values are stored under variables. Learn more about the differences between entities, variables and values here.
Test variables in the Debugger
At the moment, when you test your bot, the user should be stuck after giving the information about the ticket:
The bot does not understand yet
However, we can see that the origin and destination for the train ticket have been understood by the bot:
In the Emulator (i.e. where you test your bot on the bottom-right corner of the screen), open the Debugger
Scroll down until you see this:
The Debugger shows entities saved as variables.
So even though the sentence did give an error message, these entities are correctly recognized in the user input. This means the variable 'origin' is now saved with a variable value 'Brussels' and the variable 'destination' with the value 'Paris'. 
Still from the Emulator, if you open the NLP Result tab you can see that the intent was recognised correctly, that's great!
Let's now work on removing the error message after the user typed.
Link Book train ticket intent with a bot dialog
The error message is caused by the fact that the intent Book train ticket does not have a bot dialog linked to it. So even though it is correctly recognised, we are not telling the bot what to do when that intent is recognised. 
We can change that by adding a new Bot message: 
In the menu Bot dialogs, open the General flow
Drag and drop a Bot Message from the right-hand menu to create a new bot dialog
Drag and drop a Bot Message to your canvas to create a new bot message.
This will open the dialog. Add the following text to it: "So you want to go to {destination}, I can help you with that!" 
To reuse the variable later on in the conversation, you can put it in between curly brackets like this: {variable_name} 
When writing this message to the users, Chatlayer will automatically substitute {variable_name} with the value of the variable. If the variable is empty, an empty space will be shown.
Go to the dialog Settings and change the dialog name to 'Book train ticket'
Click Create to save your changes. Your dialog should appear like this on your canvas:
When you create a new dialog, a preview of it appears on your canvas.
Now, all that's left is to connect the intent Book train ticket to the dialog Book train ticket. To do so:
Drag and drop an Intent to your canvas
Drag and drop an intent to the bot canvas
This will open the Intent. Select Book train ticket, the intent that we've created earlier​
Under Go to, select the Book train ticket dialog
Click Create to save your changes
Your canvas should now look like this:
An intent linked to a bot dialog will be represented by an arrow on the bot canvas.
​
We have now linked the Book train ticket intent to this bot message, great job! This means that, when a user says something that triggers the Book train ticket intent, this bot message will show. 
Let's check that everything is working as expected by doing a bit of testing 👇
Step 10: Testing entities in the Emulator
Now that we have linked everything, we are ready to test if everything is configured correctly by using the Emulator.
Open the Emulator (the Test your bot tool on the bottom right)
If needed, clear the last conversation by clicking 'Clear conversation' on the top right. This starts a new conversation
Enter "I want to go to Amsterdam" and click on Submit
Open the Debugger
In the tab 'NLP Result'  you can now see if the entity was recognized correctly:
The entity 'destination' was correctly recognized
When creating new bot dialogs, you don't need to re-train the NLP
If you do not get the result as stated above, please check the following items in your bot:
If your entity is recognised by the NLP but doesn't show up in with {destination} it did not pass the threshold of 80%. Try adding that value to your entity and re-train your model, or choose another destination
If you get 'Sorry I didn't understand that', double check if your intent is linked to the Bot message and this is saved correctly.
If your intent or expression is not recognised, try re-training your NLP again.  
Now we already have a great start with linking the intent and giving a response to the expression the user says. However, we want more information from the user. Let's add more expressions and entities. 
Step 11: Multiple entities in one expression
You can add as many entities as you want to one expression. For ChooChoo, we want to more information from the user than just the destination and origin, to give a complete train-booking experience. Let's add more contextual entities!
Go to the Expressions menu
Click 'Add Expression'
Select the Book train ticket intent
Create the following expression:
I want to go from Antwerp to Brussels tomorrow at 9am in first class
And create the following entities:
origin: Antwerp
destination: Brussels
departure-date: tomorrow
departure-time: 9am
class: first class
Additional suggestions for expressions
I need to be in Paris next Thursday
I need to be in New York on Friday
I want to go to Brussels on Monday
Friday I want to go from Antwerp to Amsterdam
I want to travel in second class from Ghent to Brussel on Friday
I want to travel in first class from Antwerp to Aalst on Thursday
I like to book a first class ticket from Aalst to Brussels at nine o'clock
Tomorrow I want to go from Antwerp to Brussels on the train from 9:00 in first class
Keep in mind that NLP techniques are probabilistic in nature. When you try to capture five expressions in one sentence, it might not be able to recognise all of them correctly. As a general rule of thumb, you can start to expect reasonable results for one entity when the NLP was given at least 30 expression to learn from. Learn more about NLP best practices here.
Add more expressions with the new contextual entities to the intent. Ensure you have around 20 expressions for Book train ticket in total.
Step 12: Missing entities
Let's test out your newly created expressions:
Update the wording in the book train ticket dialog to correctly display the entities:
 I need a ticket from Antwerp to Brussels tomorrow at 9am in first class
Updating the bot dialog
If you try to update your NLP and you get an error message about 5 example entities, it means you need to add more entity values to some of your newly created entities. To do so, go to NLP > Entities > Contextual entities and make sure that that entity has at least 5 values.
Testing the expression in the emulator: @origin and @destination entities are not detected.
Uh oh, this isn't really what we expected. As you can see, not all variables were recognized correctly. So, what's the issue? Lets have a look at the NLP results in the debugger:
origin, destination, time_departure and class were found correctly, but only time_departure and class have a confidence score higher than 80%. Origin and destination score much lower, so they weren't processed as variables.
In the NLP treshold settings of our bot, we put a threshold of 80%, so anything underneath that won't be recognized correctly.  
How can you fix this issue? By adding more expressions! 
Try adding more expressions  
Add more expressions
Retrain your NLP model to see if the variables now show up correctly in the bot
Lesson recap
Your bot now has the following configuration:
3 intents with around 35 expressions in total 
5 contextual entities
A bot message, linked to the Book train ticket intent, confirming the user input in the message
You now know how to:
Create contextual entities and entity values
Use variables in a bot message 
Use multiple contextual entities in an expression
Test your input in the debugger
Not every user will give all the entities you need. In the next tutorial, you will learn how to check if a user has already provided certain information, and ask for what's missing.
2. Understanding your users
4. Asking user info through input validation
Last modified 1mo ago