CS 2001 - Fall 2006 Assignment for Dr. Hwa Assignment due by November 1, 2006. Assignment for machine translation evaluations Below the instructions is a list of translations of 30 short sentences. For each sentence, the first translation was by a machine, the next three are by people. Part 1: Read each sentence produced by the machine and compare it against the three human translations. Assign it two scores on a scale from 1-5. The first score should be for adequacy -- how closely do you think the content of the translation sentence matches the human translations? 5: It has the same meaning 4: It has added something extra or it has missed some small aspects that all the human translations mentioned 3: It mentions the same things as the human translations, but somehow got the relationships between these things all wrong 2: It uses some of the same words as the human translators but it makes no sense 1: It just makes no sense The second score should be for fluency -- how close do you think this sound like normal English? 5: It sounds like something a native English speaker might say 4: It sounds a little unnatural, but it is understandable -- like a person who learned English as a second language, for example. 3: It sounds like broken English -- most phrases are understandable, but the sentence as a whole is hard to understand. 2: It sounds like very broken English -- English-like phrases can be picked out, but the sentence makes no sense. 1: It sounds like a string of words in random permutations. Part 2: Write a program that computes the BLEU scores for these sentences. Part 3: How well did BLEU agree with you? NOTE: you may wish to compare BLEU against multiple people's scores, so you may share your scores with other people in the class *after* each of you have finished Part 1 on your own. Also, please say whose scores you used in doing this part of the assignment and how you used them. * Describe how you conducted the experiment in comparing your scores against BLEU. * Discuss why BLEU disagreed with you, * How would you modify BLEU so that it would agree with you more? -------------------------------------------------------------------------------- E11-XIN20030110.0005-1 slovakia president leaves shanghai direction hong slovakian president leaves shanghai for hong kong by air slovak president leaves shanghai for hong kong slovak president leaves shanghai for hong kong E11-XIN20030128.0160-1 chinese cultural festival in ankara opens chinese culture festival opens in ankara the chinese culture festival inaugurated in ankara chinese culture festival opens in ankara E14-XIN20030108.0124-1 us defense department cancelled two test us defense department calls off 2 missile intercept tests u . s . department of defense cancelled two missile intercept tests us defense department cancels two missile intercept tests E14-XIN20030128.0160-1 chinese cultural festival opened in ankara chinese culture festival opens in ankara the chinese culture festival inaugurated in ankara chinese culture festival opens in ankara E15-AFC20030107.0003-1 bush to rejuvenation economic plan advocated bush defends his economic stimulus package bush defends his economic stimulus plan bush defends his economic promotion plan E15-XIN20030128.0160-1 chinese cultural festival in ankara inaugurated chinese culture festival opens in ankara the chinese culture festival inaugurated in ankara chinese culture festival opens in ankara E09-XIN20030110.0005-1 slovakian president flies hong kong to shanghai slovakian president leaves shanghai for hong kong by air slovak president leaves shanghai for hong kong slovak president leaves shanghai for hong kong E11-AFC20030102.0038-1 australia reopened its embassy in manila embassy australia reopens embassy in manila australia reopened manila embassy australia reopens embassy in manila E11-AFC20030110.0014-1 guard development research center praised environmental revolution worldwatch institute praises environmental revolution worldwatch institute applauds the environmental revolution worldwatch institute hails environmental revolution E11-XIN20030108.0124-1 the defense ministry lifted twice interceptor missile us defense department calls off 2 missile intercept tests u . s . department of defense cancelled two missile intercept tests us defense department cancels two missile intercept tests E11-XIN20030110.0170-1 the united states resumed received iraqi refugees us resumes admission of iraqi refugees u . s . resumes admission of iraqi refugees us resumes to accept iraqi refugees E14-XIN20030110.0005-1 slovak president left shanghai flying hong kong slovakian president leaves shanghai for hong kong by air slovak president leaves shanghai for hong kong slovak president leaves shanghai for hong kong E15-XIN20030121.0010-6 well under worker is transport equipment . at the time , workers underground were moving equipment . at time of the accident , the workers were in the pits moving equipment . then workers underground were transporting equipments . E12-AFC20030215.0016-10 the situation just the opposite . -RRB- the situation is exactly the opposite . '' the situation is quite the contrary . '' it is on the contrary . '' E09-XIN20030121.0010-6 at that time mine shaft worker loader . at the time , workers underground were moving equipment . at time of the accident , the workers were in the pits moving equipment . then workers underground were transporting equipments . E11-AFC20030208.0077-1 krcevina silk life long hanging accidental return field content with life , hingis has no desire to return to tennis martina hingis happy with life and has no intention to return to the tennis court hingis not willing to return to tennis court with leisurely life E11-AFC20030218.0003-1 germany telecommunications industry mobilcom founder schmid declared insolvent schmid , founder of german telephone operator mobilcom , declares bankruptcy german telecommunications industry : schmid , founder of mobilcom , declares bankruptcy founder of german telecom mobilcom schmid announces bankruptcy E11-XIN20030111.0054-9 she voiced clear rich , full light . her voice is clear , mellow and sweetly resonant . her voice is sweet , loud and clear . her voice is clear and mellow . E12-AFC20030102.0038-1 the australian the re-opening office in manila embassy australia reopens embassy in manila australia reopened manila embassy australia reopens embassy in manila E12-XIN20030211.0146-4 now in iraq official inspectors about 110 . at present , there are about 110 official arms inspectors in iraq . at present , there are about 110 official inspection personnel in iraq . there are now about 110 formal inspectors in iraq . E14-AFC20030228.0014-6 analysis use only twice before their production . only the data of their first two deliveries are used for analysis . only the first two deliveries were used in the analysis . analysis were only based on their previous two-birth information . E14-XIN20030121.0010-1 lishu jixi coal mine gas explosion occurred at gas explosion in lishu coal mine in jixi a gas explosion took place at jixi 's lishu mine gas explosion occurs in lishu coal mine in jixi E14-XIN20030211.0146-4 the iraqi official inspectors about 110 people . at present , there are about 110 official arms inspectors in iraq . at present , there are about 110 official inspection personnel in iraq . there are now about 110 formal inspectors in iraq . E15-XIN20030122.0195-1 iraq war might international oil market supply enough war in iraq may lead to sufficient supply on the world oil market a war on iraq may cause abundant supply in international oil market iraqi war may lead to sufficient oil supply in international oil market E12-AFC20030228.0014-12 we believe this is a new found . we believe this is a new finding . we believe this is a new observation . we think that this is a new discovery . E09-AFC20030121.0005-1 mainland china handset user will grow will slow down growth of mobile phone users in mainland china to slow down mainland china : slower growth in number of mobile phone users growth of mobile phone users ' numbers in chinese mainland to slow down E09-XIN20030215.0094-1 japanese bell invest the development network videophone in shenyang japan 's bell to invest in developing internet videophones in shenyang japan bell invests in shenyang to develop ip visual phone japan 's bell invests to develop network visual phone in shenyang E11-XIN20030104.0061-1 belgium asked anchored dead soon samples contained fuel oil belgium demands prompt pumping of fuel oil from anchored oil tanker belgium demands broken-down oil tanker to have fuel oil pumped out urgently belgium urges anchored oil tanker to pump up fuel on board E14-XIN20030204.0083-4 warheads was carrying the goods , well preserved . the warhead was tagged and kept in good condition . the warhead was tagged and was kept in good condition . the warhead was labeled and well preserved . E15-AFC20030208.0047-1 yao ming sentiment nba the as to too fast yao ming feels nba stardom comes too fast yao ming sighed with emotion that the nba idol status comes too soon yao ming feels too fast in becoming nba idol