Combine Amazon Translate with Elasticsearch and Skedler to build a cost-efficient multi-lingual omnichannel customer care – Part 2 of 2
In the previous post, we presented a system architecture to translate text from multiple languages with AWS Translate, index this information in Elasticsearch 6.2.3 for fast search, visualize the data with Kibana, and automated sending of customized intelligence with Skedler Reports and Alerts.
In this post, we are going to see how to implement the previously described architecture.
The main steps are:
- Define API Gateway HTTP endpoints
- Build a AWS Lambda Function
- Deploy to AWS with Serverless framework
- Translate text with AWS Translate
- Index to Elasticsearch 6.2.3
- Search in Elasticsearch by language – full-text search
- Visualize, report and monitor with Kibana dashboards
- Use Skedler Reports and Alerts for reporting, monitoring and alerting
We are going to define two HTTP API methods, one to translate and index new inquiries and another one to search for them. We will use AWS API Gateway, a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale.
The core of our solution will receive the inquiry (as a string) to be translated, translate it and index the text with the translations to Elasticsearch.
We will use AWS Lambda. It lets you run code without provisioning or managing servers. You pay only for the compute time you consume – there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service – all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.
To deploy our AWS Lambda function and the API Gateway HTTP endpoints, we will use the Serverless Framework. Serverless is a toolkit for deploying and operating serverless architectures.
1. API Gateway
We are going to configure the following HTTP enpoints:
- HTTP POST /createTicket
- HTTP GET /searchTicket
The createTicket endpoint will be used to translate the text using AWS Translate and to index the document in Elasticsearch. The searchTicket endpoint will be used to search for documents in different languages. The handler for each endpoint will be an AWS Lambda function.
Below is the serverless.yml section where we have defined the two endpoints.
functions:
create:
handler: handler.create_ticket
events:
– http:
path: createTicket
method: post
search:
handler: handler.search_ticket
events:
– http:
path: searchTicket
method: get
2. AWS Lambda
Once we have defined the two endpoints, we need to write the Lambda function. We do not focus on the deploy of the function in AWS, the Serverless framework will take care of it. The function we are going to write will perform the following:
- Get the input that need be translated (the customer inquiry)
- Invoke AWS Translate and get the translations of the input
- Create the Elasticsearch document
- Index the document
def create_ticket(event, context):
body = json.loads(event[‘body’])
text = body[‘text’]
customer_code = body[‘customerCode’]
country = body[‘country’]
Detect the language of the input text
translate_client = boto3.client(‘translate’)
comprehend_client = boto3.client(‘comprehend’)
def create_ticket(event, context):
body = json.loads(event[‘body’])
text = body[‘text’]
customer_code = body[‘customerCode’]
country = body[‘country’]
target_languages = os.environ[‘AWS_TRANSLATE_SUPPORTED_LANGUAGES’].split()
dominant_language = comprehend_client.detect_dominant_language(
Text=text
)[‘Languages’][0][‘LanguageCode’]
and invoke AWS Translation
translate_client = boto3.client(‘translate’)
def get_translation(text, source_dominant_language, target_dominant_language):
return translate_client.translate_text(
Text=text,
SourceLanguageCode=source_dominant_language,
TargetLanguageCode=target_dominant_language
)[‘TranslatedText’]
Create the JSON document and index it to Elasticsearch:
es = Elasticsearch(
[os.environ[‘ELASTICSEARCH_HOST’]],
verify_certs=False
)
def index_new_document(english_text, translations, customer_code, country):
result_nested_obj = []
for key, value in translations.items():
result_nested_obj.append({“language”:key, “text”:value})
doc = {
“text” : english_text,
“language” : ‘en’,
“translations” : result_nested_obj,
“timestamp”: datetime.now(),
“customer_code”: customer_code,
“country”: country,
“ticket_number”: str(uuid.uuid4())
}
es.index(index=os.environ[‘ELASTICSEARCH_INDEX_NAME’], doc_type=os.environ[‘ELASTICSEARCH_TYPE_NAME’], body=doc)
As you may have noticed, we used environment variables. They are defined in the serverless.yml configuration file.
provider:
name: aws
runtime: python3.6
region: eu-west-1
memorySize: 1024
timeout: 300
environment:
AWS_TRANSLATE_SUPPORTED_LANGUAGES: ‘ar zh fr de pt es’ # supported AWS Translate languages
ELASTICSEARCH_HOST: ‘https://yourElasticsearchHost’
ELASTICSEARCH_INDEX_NAME: ‘customercare’
ELASTICSEARCH_TYPE_NAME: ‘ticket’
3. Deploy
We are now ready to the deploy our code to AWS.
This is how my serverless.yml looks like:
We specified the provider (AWS), the runtime (Python 3.6), the environment variables, our HTTP endpoints and the AWS Lambda function handlers.
service: awstranslate
provider:
name: aws
runtime: python3.6
region: eu-west-1
memorySize: 1024
timeout: 300
environment:
AWS_TRANSLATE_SUPPORTED_LANGUAGES: ‘ar zh fr de pt es’ # supported AWS Translate languages
ELASTICSEARCH_HOST: ‘https://yourElasticsearchHost’
ELASTICSEARCH_INDEX_NAME: ‘customercare’
ELASTICSEARCH_TYPE_NAME: ‘ticket’
functions:
create:
handler: handler.create_ticket
events:
– http:
path: createTicket
method: post
cors: true
search:
handler: handler.search_ticket
events:
– http:
path: searchTicket
method: get
plugins:
– serverless-python-requirements
custom:
pythonRequirements:
dockerizePip: false
We specified the provider (AWS), the runtime (Python 3.6), the environment variables, our HTTP endpoints and the AWS Lambda function handlers.
Deploy to AWS:
serveless deploy –aws-s3-accelerate
4. Index to Elasticsearch
Given an inquiry, we now have a list of translations. Now, we want to index this information to Elasticsearch 6.2.3.
We create a new index called customercare and a new type called ticket.
The ticket type will have the following properties:
- text: the English text
- language: the language of the text
- country: the country from where we received the inquiry
- ticket number: an ID generated to uniquely identify an inquiry
- timestamp: index time
- translations: list of the translations (text and language)
PUT /customercare
{
“mappings”: {
“ticket”: {
“properties”: {
“text”: { “type”: “text” },
“language”: { “type”: “keyword” },
“country”: { “type”: “keyword” },
“ticket_number”: { “type”: “keyword” },
“timestamp”: {“type”: “date”},
“translations”: {
“type”: “nested”,
“properties”: {
“text”: { “type”: “text” },
“language”: { “type”: “keyword” }
}
}
}
}
}
}
5. Search in Elasticsearch
Now that we indexed the data in Elasticsearch, we can perform some queries to search in a multi-lingual way.
Examples:
Full-text search through translations:
GET customercare/_search
{
“query”: {
“nested”: {
“path”: “translations”,
“query”: {
“match”: {
“translations.text”: “your text”
}
}
}
}
}
Full-text search through English text and translations:
GET customercare/_search
{
“query”: {
“bool”: {
“should”: [
{
“nested”: {
“path”: “translations”,
“query”: {
“match”: {
“translations.text”: “tree”
}
}
}
},
{
“term”: {
“text”: “tree”
}
}
]
}
}
}
Number of inquiries by a customer (full-text search):
GET customercare/_search
{
“aggs”: {
“genres”: {
“terms”: {
“field”: “customerId”
}
}
},
“query”: {
“bool”: {
“should”: [
{
“match”: {
“text”: “tree”
}
}
]
}
}
}
6. Visualize, Report, and Monitor with Kibana dashboards and search
With Kibana you can create a set of visualizations/dashboards to search for inquiries by language and to monitor index metrics (like number of translations or number of translations by customer).
Examples of Kibana dashboards:
Top languages, languages inquiries by customer and geolocation of inquiries:
inquiries count by languages and customers and top customer by language:
7. Use Skedler Reports and Alerts to easily monitor data
Using Skedler, an easy to use report scheduling and distribution application for Elasticsearch-Kibana-Grafana, you can centrally schedule and distribute custom reports from Kibana Dashboards and Saved Searches as hourly/daily/weekly/monthly PDF, XLS or PNG reports to various stakeholders. If you want to read more about it: Skedler Overview.
We have created a custom report using Skedler Report Templates that provides an overview of the tickets based on languages and countries of origin. The custom report generated by Skedler is shown below:
If you want to get notified when something happens in your index, for example, a certain entity is detected or the number of negative feedback by customers crosses a threshold value, you can use Skedler Alerts. It simplifies how you create and manage alert rules for Elasticsearch and it provides a flexible approach to notifications (it supports multiple notifications, from Email to Slack and Webhook).
We have seen how to schedule a report generation. We are now going to see how to use Skedler Alerts to get notified when something happens in our index. For example, if the number of inquiries from a specific country hits a certain threshold.
Choose the Alert Condition. For example: “the number of ticket in English must be higher than zero”.
or “the number of ticket in English coming from Italy and containing a given word must be higher than zero”.
The Skedler Alert notification in Slack looks like.
Conclusion
In this two-part blog series, we learnt how to build our own multi-lingual omni-channel customer care platform using AWS Translate, Elasticsearch, and Skedler. Let us know your thoughts about this approach. Send your comments to hello at skedler dot com.
