Measuring Quality

Can the quality of a translated text be measured using purely statistical methods? What is the definition of statistical? What do we have in mind when we speak of a method or methodology? Without doubt, care must be taken with the terminology we use due to the fact that statistical could relate to a variety of techniques which in some way or other make use of statistics. Similarly, the term computerised is bandied about and often relates to a wide spectrum of techniques.

Statistical and/or computerised (or even computer-based) techniques are most certainly the way forward for translators, translator trainers and editors/proof-readers. This is not to say that the computer will relieve us of the work we put into the translation process but will rather provide us with powerful tools that can improve the quality of translation as well as save valuable time when producing texts. Already, computer-aided human translation has given us translation memories (like Trados) that have had a huge impact on speed and quality.

Nevertheless, is it possible for computer-based tools to measure the quality of a particular text? We have seen that the analysis of language corpora can give us clues into how a text works. By looking at purely quantitative features, such as frequency lists, it is possible to assess the mechanics and style of a text. By extension, we should therefore be able to assess the quality of a translation. If translators are able to pinpoint particular linguistic elements that might correlate to the quality of a text in a particular domain we should be able to talk of a statistical-based study of translation quality.

Research into translation quality assessment seems to point to a procedural approach to the assessment of quality, however, a simpler path might be the search for specific statistical anchor points which the translation assessor could use to determine if the translation fulfils the basic functions of a text with regards to style, for example, the appropriate number of nouns, verbs, adjectives and the appropriate ratio between these parts of speech.

Statistical tools are just that. Tools. The Universal Translator so often depicted in science fiction films may well be a pipe dream and momentarily unrealistic but this does not mean we cannot push the limits of statistical and computer-based linguistics to the very edge so that not only theorists but translation practitioners can also put them to use.


11 thoughts on “Measuring Quality

  1. I just don’t think any enterprise that links computers to generating, translating, or reviewing language works or can work. Nothing I’ve seen from that field to date is even remotely promising, really, and even somethings like TMs have even done injury to the practice of translation and lowered quality in some ways. I just don’t see how numerically based quality assessment can work–mainly because “quality” depends largely on subjective elements. Even deciding what anchors to pick is subjective. Indeed, a translation that a computer might otherwise greenlight the quality of may in reality be abysmal simply because it’s too close to the original, or too clunky once read by humans. Computers just can’t deal with language properly, and they never will.

  2. Interesting work by Raf Uzar in Poland has demonstrated that it is possible to extract purely statistical information from a translation that says a lot about the ‘potential’ quality of a text. The logic is that if a particular text displays an atypical amount of a particular structure or pattern, we can assume that the text is probably poor in quality. For example, a translation that uses an above average amount of indefinite articles probably demonstrates the translator’s inability to use the article.

    In other words, texts that do not gravitate towards the average are in some way ‘atypical’ and in all probability of poor quality. Idiosyncrasies, linguistic extravagances and eccentricities are in fact exceedingly rare in language (from a statistical point of view).

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s