Translation Quality Assessment of Online Translation Systems in Translating English to Thai on Phuket Tourism

: This research aims to compare four English-to-Thai translator applications on translating the comments posted on social media. Four translator applications include Google Translate, Baidu Translate, Yandex.Translate and Microsoft/Bing Translate. The comments in this study are on three main popular attractions in Phuket, Thailand. The attractions include a temple (Wat Chalong), a viewpoint spot (Promthep Cape) and a beach (Patong beach). Fifty comments on each attraction are collected from GoogleReviews and TripAdvisor. The total of 300 comments are translated by four translators making the final translated comments of 1,200 comments. All translated comments are evaluated by three language experts on three categories including the grammar structure, the vocabulary and idiom and the overall translation. The evaluation results show that Google Translate is found to be the most suitable translator for translating English comments to Thai comments in this study at 89.33%. Google Translate receives the average suitable score of 4.20 out of 5.00. However, Google Translate is weak at word choices to generate a beautiful sentence. Baidu Translate is very superb at generating a beautiful sentence similar to human languages. However, Baidu Translate only gets a good result on a specific domain or situation. As a result, Baidu Translate is lost to Google Translate in most cases. Even though this research presents the results on Thai language and specific domains, the techniques used in this study can be applied to any language on any domain.


Introduction
Tourism industry contributes significantly to the Thai economy. Therefore, any improvement to facilitate a tourism industry will have a significant impact on the Thai economy as a whole. With rich natural resources in all regions, Thailand is a country suitable for tourism especially, Phuket. Phuket has been known as a pearl of Andaman consisting of many beautiful beaches, historical sites, cultural and religion sites. Beaches in Phuket have been certified as one of the most beautiful beaches in the world by many international awards (Polnyotee and Thadaniti, 2015). Many activities can be conducted in Phuket such as shopping, scuba diving, jet skiing and swimming.
Nowadays, the Internet becomes essential to many tourism industry revolutions and changes. Tourists' behaviors are changing due to the influence of Internet technology. With the information available on the Internet, tourists can gain more pleasure, comfort and flexibility during their trips (Chochiang, 2017). Social media takes a great part of providing such information to tourists. The tourists can search for recommendations of the place and the activities or compare the rating and reviews; finally, the tourists can also share their experiences online (Hajli, 2014). The customer feedbacks via the reviews on the social media is the key importance factor. A good feedback will attract more customers. Moreover, the tourism industries can also build their attractive social profile from their positive customer feedback (Ba and Pavlou, 2002).
Typically, the customers provide their reviews in a text-based format and most traditional statistical methods do not have an analytical capability to analyze such information (Taecharungroj and Mathayomchan, 2019). Feedback on the tourism industry is typically either negative or positive based on the customer experiences.
According to the records on TripAdvisor-one of the travel websites for travelers looking for data-there are 859 million people conducting user reviews and user opinions in 2019. While, the number of user reviews and user opinions based on https://www.statista.com only 200 million people in 2014. Therefore, the increasing of the number shows that tourists are eager to share their information online more often and such feedback can be either negative or positive according to their experiences.
Each language contains its culture of the language owner. By speaking only one language a loss of cultural diversity occurs (Müller et al., 2016). Thailand is an Asian country that has its own identity with many tourism characteristics including language, traditions and culture. Thai language is an official language in Thailand without any secondary language. Therefore, Thai people speak Thai as their main language. However, Phuket receives many international tourists. English language can be an obstacle to tourism industry workforce in Thailand because not everyone has an opportunity to be trained and has suitable English language skills. Multiple solutions exist to overcome this language barrier. Simultaneous translation is one of the promising solutions. Having a good translator will help you to learn and to practice understanding of vocabulary or sentence patterns even more. Also, a good translator can reduce the time it takes to search for information.
Thus, this research aims to analyze and compare many existing English-to-Thai language translators on the tourism industry related reviews. As a result, the employees, entrepreneurs, business owners can better understand the meaning that the foreign tourists communicate via their feedback or reviews. These data can be further analyzed in many ways such as sentiment analysis, analysis of strengths and weaknesses of the tourist spots or hotels or suggestion of improvements provided by reviews. This study is conducted as the first step to our final objective in developing an automatic review translation system for tourism industries in Thailand.
The remaining of this study is organized as follows. Next, the review of literature is given. Then, the design choice and the tools of the proposed application is descripted. The resulting application is then discussed. Last, the conclusion is given.

Quality of Translation Assessment
Translation is an action to convey the meaning of messages from one language to another. Figure 1 shows the translation process. Translation is an understanding of the meaning of the text and later creating an equivalent message; "translation" is for communicating the same message in other languages (Kull, 1998).
There exist many important criteria for determining the quality of the translation (Intakosum, 2014) such as the emphasis on the meaning in which the translation must convey the meaning and the same flavor as the original. The interpretation of the meaning must be complete and unambiguous. In most cases, translation errors are related to grammar structure errors, in which the translator does not understand what the sentence structures or words do or how they relate to each other. Furthermore, any mistake in vocabulary and expression used in which the translator is not careful then polysemy or homographic can also be observed.

Translation Technologies
The development of computerized translation tools has not stopped while the cloud-based translation technologies are emerging. With cloud-based translation technologies, a complex and highly digitized environment for the latest translation technology can be accessible without the need to own an expensive computer with a fast processor and a large amount of disk space (Muegge, 2012). A software that allows translators to translate written text from one natural language (original language) into a text in another natural language (target language) is known as a translation tool (Kastberg and Andersson, 2012). Existing translator tools that are well known to translate English language to Thai language include Baidu Translate; Microsoft/Bing Translate; Google Translate; Yandex.Translate.  (Door43, 2020) Baidu Translate is a software application search engine service provider from China. Baidu machine translation system was released in June 2011. Currently, Baidu supports 27 languages on multiple platforms including personal computers, mobile devices and a set of APIs for developers (He, 2015). The underlying technology used by Baidu is deep learning technologies. The machine translation tool processes approximately 100 million requests every day. Similar to Google Translate, Baidu translate supports Optical Character Recognition (OCR) and Baidu can also "translate" the object. By pointing the camera at a plant, the plant will be labeled as "plant" in the user preferred language (Sun, 2014 (Martin, 2017) and Microsoft/Bing Translate has been found to produce more errors in its translation than that of Google Translate. Also, Microsoft/Bing Translate was reported to have an issue in translating specialized text. On the other hand, the comparison of Google Translate and Microsoft/Bing Translate in translating English to Hindi and vice versa in (Dhakar et al., 2013) found that Microsoft/Bing Translate produced a better translation quality.
Google Translate was published at the end of 2008 by Google. The development of PHP and AJAX allows Google Translate to produce a media of translation called translator. "Google Translate" is a medium of online translation service. Google Translate is supporting more than 50 languages (Ulfah, 2015). The underlying technology of Google Translate is also statistical machine translation which is a process of seeking patterns in a large number of texts in order to find a sentence close to a human language. The technology is based on the training of statistical models from the large corpora of human translation. This technology has an advantage of fast training if there is a corpora available when comparing to the rule-based system. The technology has been known to be quite good at solving vocabulary problems (Vidhayasai et al., 2015). Google Translate has been shown to be feasible in translating medical articles in many cases and the process is not a resource-intensive (Balk et al., 2012). However, the translation quality relied heavily on the article's original language. In particular, Roman languages have a higher level of agreement than that of Asian languages and Hebrew.
In early 2011, Yandex implemented a machine translation system. The underlying technology of Yandex.Translate is not based on language rules. Yandex.Translate also uses statistical method and supports major European languages. Yandex.Translate is also supporting Thai language via Yandex Translation API. Yandex.Translate supports both free-form texts and the whole web pages.

Methodology
The performance of four translators including Google Translate (denoted Translator_A), Microsoft/Bing Translate (denoted Translator_B), Yandex.Translate (denoted Translator_C) and Baidu Translate (denoted Translator_D), on translating English language reviews to Thai language reviews are evaluated in this study.

Workloads
To evaluate the four translators, the datasets of Phuket main attractions were collected from GoogleReviews and TripAdvisor. GoogleReviews and TripAdvisor are online-based websites that allow users to review their experience from the real attractions. Both GoogleReviews and TripAdvisor are big and popular travel websites for tourists to visit in order to search for data and to plan their trips. It had been shown in numerous previous studies on traveler behavior that reviews clearly influenced travelers' booking decisions (Xie et al., 2016;Nilashi et al., 2018).
Three domains of attractions including Temple (Wat Chalong), Viewpoint (Promthep Cape) and Beach (Patong beach) are selected in this study. Each domain is very distinct and unique in itself. Fifty comments were collected from each source for each attraction domain as shown in Table 1. Only English comments were used in this study. The data sampling method in this study is selecting fifty comments dated back from January 2020 until reaching 50 comments on each attractions and each source; only English comments were collected. As a result, the reviews of Temple were collected from August 2019 to January 2020. The reviews of Viewpoint were collected from March 2019 to January 2020 and the reviews of Beach were collected from September 2019 to January 2020.
According to the dataset characteristics shown in Table 1, the average number of sentences and words of the whole dataset is at 174.50 sentences and 2,126.83 words. While, the average number of sentences and words in each review is at 3.49 sentences and 42.54 words. For the characteristics of TripAdvisor datasets, the average number of sentences and words is at 178 sentences and 2,204.67 words. For the characteristics of GoogleReviews datasets, the average number of sentences and words is at 171 sentences and 2,049 words. Figure 2 shows the evaluation method in this study. There are fifty comments of each attraction from two sources making the total comments of 300 English comments. Each of the four translators translates all 300 English comments in Thai comments resulting in 300 translated Thai comments for each translator. Thus, the total Thai translated comments from all four translators is 1,200. The translation performance of all four translators are evaluated by three language experts. Each expert is a respective faculty member of a Thai University. Each expert has expertise in Educational and Applied Linguistics (English as an International Language).

Evaluation Methods
Each expert must evaluate all 1,200 Thai translated comments. For fairness reasons, the real identity of the translator is not known to the experts. Thus, the experts will be given the comments with the information of the alias names of the translators. Thus, Google Translate alias name is Translator_A; Microsoft/Bing Translate alias name is Translator_B; Yandex.Translate alias name is Translator_C; Baidu Translate alias name is Translator_D. Therefore, each expert will evaluate the same set of comments-1,200 Thai translated comments.
The expert must evaluate the translated Thai comments on three categories, including grammar structure, vocabulary and idiom and overall translation. Each category consists of several qualities. These qualities are compiled from translation issues and good features of translation assessments in the literature (Intakosum, 2014;Chanwaiwit, 2016). The evaluations include accuracy, appropriateness and cohesion and focus on examining common grammar structure, vocabulary and idiom problems.
The first category is the grammar structure which consists of 3 qualities including (1) meet the grammar structure completely, (2) has consistency of words and has a coherent relationship in each sentence and (3) be able to communicate appropriately. These three qualities will enable the researcher to evaluate the grammatical structure covering word consistency, interpretation and the complete grammar structure. The second category is the vocabulary and idiom which consists of 2 qualities including (1) choose a correct vocabulary and (2) is clear and complete content. These two qualities will enable the researcher to know that the chosen vocabulary matches the meaning and complete with no missing parts.
The last category is the overall translation which consists of 2 qualities including (1) is the same or similar to the original and (2) give a feeling, mood, pleasure in reading equal to the original. Both qualities will focus on the translation that must still have the same meaning and clarity as the original.
The evaluation score of each expert on each comment is on a scale of 1 to 5 with the following meaning: 1. Means the text is clear and does not require corrections 2. Means the text contains minor errors 3. Means the meaning of the text is reduced to «gist» but is understandable 4. Means the text is barely comprehensible 5. Means the text is incomprehensible

Results and Discussion
The comments in this study were collected from three attractions in Phuket including a temple (Wat Chalong), a viewpoint (Promthep Cape) and a beach (Patong beach). Each place consists of two dataset from two sources including GoogleReviews and TripAdvisor. Each data set contains 50 comments. For each comment, there are four translated comments resulting from four translators (i.e., Translator_A, Translator_B, Translator_C and Translator_D). There are three experts (P1, P2 and P3). All three experts must evaluate all 1,200 translated comments on three categories including grammar structure, vocabulary and idiom and overall translation. Then, the expert will select one of the translators in which he/she found to best translate the comments.
During the evaluation process, each expert will give a score on each translator ability in translating each comment. The score is ranged between 1 and 5. The score will be given in three categories, include grammar structure, vocabulary and idiom and overall translation. Moreover, the expert will also give the detail analysis on his/her decision. To analyze further, the expert choice on each comment is then studied. To calculate the best achievable performance to be used as the baseline for analysis, the performance of the best translator on each comment is presented as the best achievable performance. However, the winning translator must be selected. Thus, the winning translator for each comment is selected based on the voting of three expert choices. That is, if two or more experts select Translator_A for that comment, Translator_A will be selected as the winning translator for that comment. The results of the winning translator for each data set is also presented and discussed.

Evaluation Results on Temple Dataset
For the temple (Wat Chalong) dataset, the average number of sentences for each comment is 3.44 while the average number of words for each comment is 41.95. Mostly, the comments are related to the beauty of Buddha statue, the temple history and the dress code of the visitors. Table 2 shows the average score of the best achievable performance on three categories given by three experts on the temple dataset. According to the results shown in Table  2, the score on the vocabulary and idiom is lower than the other two categories. The reason behinds this result is that the temple dataset is related to a specific place which has a specific set of vocabularies. For example, "Moderate dress is required" has been translated by all translators to "สวมใส่ ชุ ดปานกลาง" (suamsai chut panklang) meaning "wear medium set" which is clearly incorrect. Table 3 shows the average score of the best achievable performance on each category of temple datasets. According to the results shown in Table 3, the translated comments mostly met the grammar structure completely. Also, the translated comments have a consistency of words and have a coherent relationship in each sentence. As a result, the translated sentence gives the correct meaning. For example, "Wat Chalong is one of the most impressive and most popular temples in Phuket." is translated to "วั ดฉลองเป็ นหนึ ่ งในวั ดที ่ น่ าประทั บใจและเป็ นที ่ นิ ยมมากที ่ สุ ดในภู เก็ ต" (wat chalong pen nueng nai wat thi na prathapchai lae pen thi niyom mak thisut nai phu ket) which is "Chalong Temple is one of the most impressive and popular temples in Phuket.". As can be seen that the translation is correct and complete. The translated sentence has a consistency of words including various connecting words. The reader can understand the translated sentence. Table 4 shows the results of each expert on his/her selection of the translator on the temple datasets. There are 50 English comments from GoogleReviews and TripAdvisor each. The expert must select the best translator for each of the 50 comments. According to the results shown in Table 4, the results of all three experts are quite unanimous. That is, Translator_C is not selected by any expert. All three experts select Translator_A the most followed by Translator_D and Translator_B. For GoogleReviews dataset, the experts select Translator_A 72.00% (i.e., 108 comments), Translator_D 24.67% (i.e., 37 comments) and Translator_B 3.33% (i.e., 5 comments). For TripAdvisor dataset, the experts select Translator_A 89.33% (i.e., 134 comments), Translator_D 8.67% (i.e., 13 comments) and Translator_B 2.00% (i.e., 3 comments). In total, Translator_A is selected for 242 comments or 80.67% which is significantly larger than the other three translators.
In conclusion, the number of comments winning by Translator_A-the winning translator selected by the experts-for the GoogleReviews temple dataset is 38 out of 50 comments or 76%. The number of comments winning by Translator_A for the TripAdvisor temple dataset is 48 out of comments or 96%. The number of comments wining by Translator_A for both datasets 86 out of 100 comments or 86%.
Furthermore, there are two comments in which Translator_A is not selected due to the translation error made by Translator_A. Both comments are perfectly translated by Translator_D. For example, "Completely free of cost!" is being translated to "เสี ยค่ าใช ้จ่ ายโดยสิ ้ นเชิ ง" (sia khachaichai doisinchoeng) by Translator_A which means "waste the cost altogether". On the other hand, Translator_D translates the sentence to "ฟรี ทั ้ งหมด" (fri thangmot) which means "free of charge". As can be seen that Translator_D is correctly translating the sentence.

Evaluation Results on Viewpoint Dataset
For the viewpoint (Promthep Cape) dataset, the average number of sentences for each comment is 3.53 while the average number of words for each comment is 45.14. Mostly, the comments are related to landscape, sunset and views. Table 5 shows the average score of the best achievable performance on three categories given by three experts on viewpoint dataset. Similar to the temple dataset, the average scores on grammar structure category is the highest while the average score on the vocabulary and idiom is the lowest. However, the score on the vocabulary and idiom of the viewpoint dataset is higher than that of the temple dataset. The reason behind this result is that the viewpoint dataset do not contain as much specific vocabularies. Table 6 shows the average score of the best achievable performance on each category of the viewpoint dataset. According to the results shown in Table 6, the translated comments mostly met the grammar structure completely. Also, the translated comments are the same or similar to the original comments. The average scores on the viewpoint dataset are higher than their counterpart on the temple dataset ( Table 2). The reason behind this result is that the vocabularies of the viewpoint dataset are mostly general vocabularies. Unlike the temple dataset, there is no specific set of vocabularies in the viewpoint dataset. Mostly, the comments describe the beauty of nature which is general. Table 7 shows the results of each expert on his/her selection of the translator on the viewpoint dataset. There are 50 English comments from GoogleReviews and TripAdvisor each. The expert must select the best translator for each of the 50 comments. According to the results shown in Table 7, the results of all three experts are quite unanimous. That is, all three experts do not select Translator_C. Similar results are observed here. That is, all three experts select Translator_A the most followed by Translator_D and Translator_B. For the GoogleReviews dataset, the experts select Translator_A 78.00% (i.e., 117 comments), Translator_D 14.67% (i.e., 22 comments) and Translator_B 7.33% (i.e., 11 comments). For the TripAdvisor dataset, the experts select Translator_A 81.33% (i.e., 122 comments), Translator_D 16% (i.e., 24 comments) and Translator_B 2.67% (i.e., 4 comments). In total, Translator_A is selected for 239 comments or 79.67% which is significantly larger than the other three translators.       In conclusion, the number of comments winning by Translator_A-the winning translator selected by the experts-for the GoogleReviews viewpoint dataset is 45 out of 50 comments or 90%. The number of comments winning by Translator_A for the TripAdvisor viewpoint dataset is 42 out of comments or 84%. The number of comments wining by Translator_A for both datasets is 87 out of 100 comments or 87%.
Furthermore, there are two comments in which Translator_A is not selected due to the translation error made by Translator_A. Both comments are perfectly translated by Translator_D. For example, "but never stopped us from squeezing to the cliff front to have a good view" is being translated to "แต่ ไม่ เคยหยุ ดเราจากการบี บหน้ าผาเพื ่ อให ้ได ้มุ มมองที ่ ดี " (tae mai khoei yut rao chak kan bip napha phuea hai dai mum mong thi di) by Translator_A which means "But never stopped us from squeezing the cliff for a good view". On the other hand, Translator_D translates the sentence to "แต่ ไม่ เคยหยุ ดเราจากการเข ้าถึ งหน้ าผาเพื ่ อเพลิ ดเพลิ นกั บทั ศ นี ยภาพที ่ สวยงาม" (tae mai khoei yut rao chak kan khaothueng napha phuea phloetphloen kap thatsaniyaphap thi suai-ngam) which means "But never stopped us from reaching the cliffs to enjoy the beautiful scenery". As can be seen that Translator_D is correctly translating the sentence with a set of beautiful words.
Another example, "Very honestly, it's a brilliant sunset point and that's about it" is being translated to "ตรงไปตรงมามากมั นเป็ นจุ ดชมพระอาทิ ตย ์ตกที ่ สวยงามและที ่ เ กี ่ ยวข ้องกั บมั น" (trong pai trong ma mak man pen chut chom phra athit tok thi suai-ngam lae thi kiaokhong kap man) by Translator_A which means "Very straightforward, it was a beautiful sunset point and related to it". While, Translator_D translates the sentence to "บอกตามตรงมั นเป็ นจุ ดที ่ พระอาทิ ตย ์ตกที ่ ยอดเยี ่ ยมแค่ นั ้ น" (bok tam trong man pen chut thi phra athit tok thi yotyiam khaenan) which means "To be honest, it was just that wonderful sunset point". Even Though Translator_A is correctly translated the sentence but Translator_D produces more pleasure in reading the sentence than that produced by Translator_A.

Evaluation Results on Beach Dataset
For the beach (Patong Beach) dataset, the average number of sentences for each comment is 3.50 while the average number of words for each comment is 40.51. Mostly, the comments are related to the beach, location and activities, with both positive and negative comments. Table 8 shows the average score of the best achievable performance on three categories given by three experts on the beach dataset. One interesting finding from the beach data is that the average score on GoogleReviews dataset is higher than that of the TripAdvisor dataset. The cause of this result is that the comments on TripAdvisor dataset contain incomplete sentences and slangs. Table 9 shows the average score of each category in detail for both beach datasets. According to the results shown in Table 9, the translated comments mostly met the grammar structure completely. However, the word choice seems to trouble the translators in this case. The main reason is that many words can have various meanings. Selecting the vocabularies that are suitable for the context is the key to success. Moreover, some words may be in a metaphorical manner. For example, all translators translate "The water isn't really clear" to "น ้ าไม่ ชั ดเจน" (nam mai chatchen) which means "water is not obvious". Another example, "Local sellers are heavy talkers and sticky" is being translated to "ผู ้ ขายในพื ้ นที ่ เป็ นนั กพู ดหนั กและเหนี ยว" (phukhai nai phuenthi pen nak phut nak lae niao) by Translator_A which means "Local vendors are heavy speakers and gummy". While the sentence is being translated to "ผู ้ ขายท ้องถิ ่ นเป็ นคนช่ างพู ดเหนี ยว" (phukhai thongthin pen khon chang phut niao) by Translator_D which means "Local seller is a talkative gummy person".        Table 10 shows the results of each expert on his/her selection of the translator on the beach dataset. There are 50 English comments from GoogleReviews and TripAdvisor each. The expert must select the best translator for each of the 50 comments. According to the results shown in Table 10, the results of all three experts are quite unanimous. That is, all three experts do not select Translator_C. All three experts select Translator_A the most followed by Translator_D and Translator_B. For GoogleReviews beach dataset, the experts select Translator_A 86.00% (i.e., 129 comments), Translator_D 12.67% (i.e., 19 comments) and Translator_B 1.33% (i.e., 2 comments). For TripAdvisor dataset, the experts select Translator_A 89.33% (i.e., 134 comments), Translator_D 10.00% (i.e., 15 comments) and Translator_B 0.67% (i.e., 1 comments). In total, Translator_A is selected for 263 comments or 87.67% which is significantly larger than the other three translators.
According to the results shown in Table 10, Translator_A has been selected for almost 100% due to its accuracy in translating the comments in the beach datasets. Especially, Translator_A gets almost all short sentences translated correctly.
In conclusion, the number of comments winning by Translator_A-the winning translator selected by the experts-for the GoogleReviews beach dataset is 45 out of 50 comments or 90%. The number of comments winning by Translator_A for the TripAdvisor beach dataset is 50 out of 50 comments or 100%. The number of comments wining by Translator_A for both datasets is 95 out of 100 comments or 95%.

Evaluation Results on the Whole Dataset
In conclusion, Translator_A is winning for 268 out of 300 comments or 89.33%. For GoogleReviews dataset, Translator_A is winning for 128 out of 150 comments of 85.33%. For TripAdvisor dataset, Translator_A is winning for 140 out of 150 comments or 93.33%. Table 11 shows the performance of Translator_A against the performance of the best achievable performance. According to the results shown in Table  11, the performance of Translator_A is very close to the performance of the best answer as can be seen that Translator_A has been selected for 72 to 89.33% of the times. The outstanding quality of Translator_A is that Translator_A produces the resulting translated comments that meet the grammar structure completely with clear and complete content.
On the GoogleReviews temple dataset, Translator_A has been selected for only 108 out of 150 which is the least among all six datasets. The reason behind this result is that Translator_D outperforms Translator_A in producing a result that is a natural language and has more discretion in context. For example, "Very clean and beautiful temple. people follow discipline here" is translated to วั ดสวยสะอาดมาก ผู ้ คนที ่ นี ่ ปฎิ บั ติ ตามระเบี ยบวิ นั ย (wat suai sa-at makphukhon thi ni pati bat tam rabiap winai) by Translator_D which means "Beautiful temple, very clean people here comply with discipline" while Translator_A translates the sentence to "วั ดที ่ สะอาดและสวยงามมาก คนติ ดตามวิ นั ยที ่ นี ่ " (wat thi sa-at lae suai-ngam makkhon tittam winai thi ni) which means "Very clean and beautiful temple people trace discipline here". Antoher example, "Nice Peachful Temple" is translated to "วั ดที ่ สวยงามและเงี ยบสงบ" (wat thi suai-ngam lae ngiap sangop) by Translator_D which means "Beautiful and quiet temple" while Translator_A translates the sentence to "วั ดสั นติ ที ่ ดี " (wat santi thi di) which means "peace temple that is good". Translator_D can translate some comments to natural human language results. Therefore, Translator_D receives a very high 5 out of 5 score on some comments in the GoogleReviews temple dataset.
In total, there are six datasets. Each dataset contains 50 comments. Thus, there are 300 comments. By counting individual expert result, Translator_A has been selected 744 times or 82.67%; Translator_B has been selected 26 times or 2.89%; Translator_D has been selected 130 times or 14.44%; no expert selects Translator_C. The quality of the best answer is at 4.21 while the quality of Translator_A answer is at 4.20 which is very similar. As a result, it can be concluded that Translator_A outperforms other translators in this study. In this study, the experts do not know the identity of the translator for a fair judgement on the results.
According to the results shown in this study, Yandex.Translate (denoted Translator_C) is not suitable for translating the comments. The resulting sentence translated by Yandex.Translate is either incomplete or incorrect grammar structure. For example, "good sand" has translated to "สิ นค ้าดี " (sinkha di) which means "Good product". Another example, "Clean beach" has been translated to "สะอาดชายหาด" (sa-at chaihat) meaning "beach clean" which is incorrect sentence structure in Thai. Or "Peaceful place to spend some time" has been translated to "สงบสถานที ่ ต ้องใช ้เวลาหน่ อย" (sa-ngop sathanthi tong chai wela noi) meaning "Peaceful place, need a little time".
For Microsoft/Bing Translate (denoted Translator_B), some sentences have been translated correctly. However, some resulting sentences translated by Microsoft/Bing Translate are incomplete or have incorrect meaning. For example, "Near Bangla market and good" has been translated to "ใกล ้ตลาดบางลา" (klai talat bang la) which is correct but "and good" is missing from the result. Another example, "Lovely beach close to many amenities. Highly recommended" has been translated to "มี สิ ่ งอ านวยความสะดวกมากมาย 9.6" (mi sing amnuai khwam saduak makmai 9. 6) which means "There are many facilities 9.6" which is clearly incorrect.
Baidu Translate (denoted Translator_D) receives the second rank in this study, especially on the temple datasets. Baidu Translate must improve the translation of specific names such as Wat Chalong has been translated to Wat-Cha-ra which is a common Thai male name. Moreover, Baidu Translate is not consistent on the results. Some comments are well translated while some comments are completely incorrect. The good qualities of Baidu Translate include natural language, good word choices and the same expressions as the original. An example of some good results include "Very nice beach" has been translated to "ชายหาดที ่ สวยงามมาก" (chaihat thi suai-ngam mak) which means "Very beautiful beach".
The good qualities of Google Translate (denoted Translator_A) which is the winning translator in this study include translating all words, choosing standard words, polite languages, preserving the English sentence structure and translating the location name correctly.

Conclusion
In this research, four popular translators (Google Translate, Baidu Translate, Yandex.Translate and Microsoft/Bing Translate) are compared in translating English to Thai. The data used in the experiment comes from comments on social media related to the 3 major tourist attractions in Phuket, Thailand. This research has been evaluated by 3 language experts from a university in Thailand. The assessment covers 3 major topics including the grammar structure, the vocabulary and idiom and the overall translation. The evaluation results show that Google Translate is found to be the most suitable translator for translating English comments to Thai comments in this study at 89.33%. For GoogleReviews datasets, Google Translate has been selected 85.33% while Google Translate has been selected 93.33% for TripAdvisor datasets. The overall performance of the best translator on the grammar structure, the vocabulary and idiom and the overall translation is at 4.31, 4.05 and 4.21, respectively. Even though this research presents the results on specific languages and domains, the techniques used in this study can be applied to any language on any domain. Our future work is analysis of word cut translation to extract the important themes in tourism in order to analyze the strengths and any improvement for the attraction. Since Phuket is a place that is popular among foreigners, another future work direction includes a comparison of these translators for other foreign languages such as Chinese, Russian and French to Thai language in order to find a suitable translator.