Spot the difference!

I don’t know if you can recall these games from when you were a kid?  I used to spend hours trying to find all the differences between the image on the left and the one on the right.  I never once thought how that might become a useful skill in later life… although in some cases it’s a skill I’d rather not have to develop!

You may be wondering where I’m going with this so I’ll explain.  Last weekend the SFÖ held a conference in Umeå, Sweden… I wasn’t there, but I did get an email from one of my colleagues asking how you could see what changes had been made in your bilingual files as a result of post-editing Machine Translation.  The easy answer of course is to do the post-editing with your track changes switched on, then it’s easy to spot the difference.  That is useful, but it’s not going to help with measurement, or give you something useful to be able to discuss with your client.  It’s also not going to help if you didn’t work with tracked changes in the first place because you’d need some serious spot the difference skills to evaluate your work!

Now, I’ve discussed this sort of thing, how to be productive with MT and ways to measure it, in great detail in the past in these articles:

• There’s more than one way to skin a Cat
• Solving the post-edit puzzle
• Qualitivity… measuring quality and productivity

So why talk about them again?  Well, I think in the past I went into so much detail I think I probably frightened many people off and they may not realise just how simple this can be.  So I’m going to very quickly look at how to measure your effort when you do post-edit work using two different plugins and explain the difference between them.  To do this I have a simple text in German which I ran through a machine translation engine and then post-edited it without track changes on.  I then have two versions for a bit of “spot the difference” where you can see from the AT status that I edited all but the first two segments:

I’ll start with Post-Edit Compare.

Post-Edit Compare

This plugin is available on the SDL AppStore and the developer, Patrick Hartnett, has versions for Studio 2009, 2011, 2014 and 2015.  I’m only going to look at 2015 where it also has the additional advantage of being nicely integrated into the product as you’ll see in the video.  It’s only about 7-minutes long and worth a watch to see how the plugin can be used to compare the differences between the two files and provide an analysis to summarise the effort you had to use:

Next I’ll take a quick look at Qualitivity.

Qualitivity

This plugin is also available on the SDL AppStore and the developer, also Patrick Hartnett, has versions for Studio 2014 and 2015.  I’m only looking at Studio 2015 again, but the principle is exactly the same for both versions.  The video is also about 7-minutes long and shows a small part of the features that are actually available with this tool, specifically related to the comparison we’re looking for here.  Of the two plugins this is my favourite simply because it is completely automated and you don’t actually have to compare anything.  It is tracking everything you do instead of simply comparing files, although this does mean there are interesting differences between the tools and why you would choose one over the other (see the summary at the end):

Summary

Both of these plugins do a really good job of quantifying the changes you have had to make to turn a machine translated file into a polished and customer ready target language file.  But there are differences between them that are worth knowing so you can use the one most appropriate for your needs.  I’ll just discuss these a little here to round off the article:

1. If you prepare machine translated files and send them off for post-editing then Post-Edit Compare may be the easiest plugin to use.  You can save each version of the project as a new Post-Edit Project and easily compare between any two versions you like.  You also have the ability to easily restore your Project to whichever version you like.
2. Qualitivity does not compare two files.  It tracks everything you do while post-editing and allows the post-editor to generate extremely rich reports that can be used for post-edit analysis.  But as a Project Manager you would need to ask the Post-Editor for these reports in order to be able to see the changes yourself.
3. If you are doing academic research, or analysing Post-Editing techniques then Qualitivity is the tool for you.
   1. It records the data in the background quietly without interfering with the work of the post-editor at all
   2. It tracks everything the Post-Editor is doing within Studio, and only within Studio, and this can be very useful.  Both tools calculate a similar PEM (Post Edit Modification) score but Qualitivity will show you what steps the Post-Editor had to make in order to get to the post-edited version, for example:
      
      If you look closely you’ll see the differences.  Qualitivity is actually track changes based on what you actually did and how you edited the segment.  Post-Edit Compare shows you the changes, but without any knowledge of how the changes were actually made keystroke by keystroke.  Qualitivity can capture these keystrokes and how long it took to make them!

In a nutshell these are the differences as far as quantifying Post-Editing in its simplest form is concerned.  Both tools are excellent and very simple to use.  I hope this article and the videos demonstrate this, but if you want more details don’t forget you can always investigate these articles to learn more:

• Post-Edit Compare : Solving the post-edit puzzle
• Qualitivity… measuring quality and productivity