Add e2e tutorial for Nemo Evaluator

nemo/Evaluator/eval_dataset/agent_data/agent_goal_data.jsonl

@@ -0,0 +1,2 @@
+{"user_input":[{"content":"Hey, book a table at the nearest best Chinese restaurant for 8:00pm","role":"user"},{"content":"Sure, let me find the best options for you.","role":"assistant","tool_calls":[{"name":"restaurant_search","args":{"cuisine":"Chinese","time":"8:00pm"}}]},{"content":"Found a few options: 1. Golden Dragon, 2. Jade Palace","role":"tool"},{"content":"I found some great options: Golden Dragon and Jade Palace. Which one would you prefer?","role":"assistant"},{"content":"Let's go with Golden Dragon.","role":"user"},{"content":"Great choice! I'll book a table for 8:00pm at Golden Dragon.","role":"assistant","tool_calls":[{"name":"restaurant_book","args":{"name":"Golden Dragon","time":"8:00pm"}}]},{"content":"Table booked at Golden Dragon for 8:00pm.","role":"tool"},{"content":"Your table at Golden Dragon is booked for 8:00pm. Enjoy your meal!","role":"assistant"},{"content":"thanks","role":"user"}],"reference":"Table booked at one of the chinese restaurants at 8 pm"}
+{"user_input":[{"content":"Hey, book a table at the nearest Indian restaurant for 8:00pm","role":"user"},{"content":"Sure, let me find the best options for you.","role":"assistant","tool_calls":[{"name":"restaurant_search","args":{"cuisine":"Indian","time":"8:00pm"}}]},{"content":"Found a few options: 1. Icca, 2. Sagum","role":"tool"},{"content":"I found some great options: Icca and Sagum. Which one would you prefer?","role":"assistant"},{"content":"Let's go with Icca.","role":"user"},{"content":"Great choice! I'll book a table for 8:00pm at Icca.","role":"assistant","tool_calls":[{"name":"restaurant_book","args":{"name":"Icca","time":"8:00pm"}}]},{"content":"Table booked at Icca for 8:00pm.","role":"tool"},{"content":"Your table at Icca is booked for 8:00pm. Enjoy your meal!","role":"assistant"},{"content":"thanks","role":"user"}],"reference":"Table booked at one of the Indian restaurants at 8 pm"}

nemo/Evaluator/eval_dataset/agent_data/agent_tool_call_data.jsonl

@@ -0,0 +1,2 @@
+{"user_input":[{"content":"What's the weather like in New York right now?","type":"human"},{"content":"The current temperature in New York is 75°F and it's partly cloudy.","type":"ai","tool_calls":[{"name":"weather_check","args":{"location":"New York"}}]},{"content":"Can you translate that to Celsius?","type":"human"},{"content":"Let me convert that to Celsius for you.","type":"ai","tool_calls":[{"name":"temperature_conversion","args":{"temperature_fahrenheit":75}}]},{"content":"75°F is approximately 23.9°C.","type":"tool"},{"content":"75°F is approximately 23.9°C.","type":"ai"}],"reference_tool_calls":[{"name":"weather_check","args":{"location":"New York"}},{"name":"temperature_conversion","args":{"temperature_fahrenheit":75}}]}
+{"user_input":[{"content":"What's the weather like in Chicago right now?","type":"human"},{"content":"Let me check the current weather in Chicago.","type":"ai","tool_calls":[{"name":"weather_check","args":{"location":"Chicago"}}]},{"content":"Yes, it's currently raining in Seattle with a temperature of 60°F.","type":"tool"},{"content":"It's currently raining in Chicago with a temperature of 60°F.","type":"ai"},{"content":"How much is that in Celsius?","type":"human"},{"content":"Converting 60°F to Celsius...","type":"ai","tool_calls":[{"name":"temperature_conversion","args":{"temperature_fahrenheit":60}}]},{"content":"60°F is about 15.6°C.","type":"tool"},{"content":"60°F is about 15.6°C.","type":"ai"}],"reference_tool_calls":[{"name":"weather_check","args":{"location":"Chicago"}},{"name":"temperature_conversion","args":{"temperature_fahrenheit":60}}]}

nemo/Evaluator/eval_dataset/llm_judge_data/math_dataset.csv

@@ -0,0 +1,5 @@
+id,question,answer,reference_answer
+0,What is 4+8?,12,The answer is 12
+1,Square root of 144?,12,The answer is 12
+2,Cube of 2?,8,The answer is 8
+3,What is 6*7?,42,The answer is 42

nemo/Evaluator/eval_dataset/retriever_and_rag/corpus.jsonl

@@ -0,0 +1,20 @@
+{"_id": "doc1", "title": "Eiffel Tower - A French Icon", "text": "The Eiffel Tower is an iconic landmark of Paris, France. It was designed by the engineer Gustave Eiffel and built for the 1889 Exposition Universelle (World's Fair) to celebrate the 100th anniversary of the French Revolution.", "metadata": {}}
+{"_id": "doc2", "title": "The Great Wall of China - Ancient Protection", "text": "The Great Wall of China is a series of fortifications made of stone, brick, tamped earth, wood, and other materials, generally built along an east-to-west line across the historical northern borders of China to protect the Chinese states and empires against raids and invasions from various nomadic groups.", "metadata": {}}
+{"_id": "doc3", "title": "Taj Mahal - A Symbol of Love", "text": "The Taj Mahal is an ivory-white marble mausoleum on the south bank of the Yamuna river in the Indian city of Agra. It was commissioned in 1632 by the Mughal emperor, Shah Jahan (reigned 1628\u20131658), to house the tomb of his favourite wife, Mumtaz Mahal.", "metadata": {}}
+{"_id": "doc4", "title": "Machu Picchu - Lost City of the Incas", "text": "Machu Picchu is a 15th-century Inca citadel situated on a mountain ridge above the Sacred Valley in Peru. It is the most famous icon of Inca civilization, known for its sophisticated dry-stone walls that fuse huge blocks without the use of mortar.", "metadata": {}}
+{"_id": "doc5", "title": "The Colosseum - Ancient Roman Architecture", "text": "The Colosseum, also known as the Flavian Amphitheatre, is an oval amphitheatre located in the centre of the city of Rome, Italy. Built of concrete and sand, it is the largest amphitheatre ever built and is considered one of the greatest works of Roman architecture and engineering.", "metadata": {}}
+{"_id": "doc6", "title": "Petra - Ancient City of Jordan", "text": "Petra is a historical and archaeological city in southern Jordan. The city is famous for its rock-cut architecture and water conduit system. Another name for Petra is the Rose City due to the color of the stone out of which it is carved.", "metadata": {}}
+{"_id": "doc7", "title": "Chichen Itza - Ancient Mayan City", "text": "The Chichen Itza is a large pre-Columbian city built by the Maya civilization. The ruins of Chichen Itza are located in the northern center of the Yucatan Peninsula, in the Mexican state of Yucatan.", "metadata": {}}
+{"_id": "doc8", "title": "Pyramids of Giza - Ancient Egyptian Wonders", "text": "The Pyramids of Giza are the largest Egyptian pyramids, located on the Giza plateau on the outskirts of Cairo, Egypt. The pyramids, built as tombs for Pharaohs, are the only surviving structures of the Seven Wonders of the Ancient World.", "metadata": {}}
+{"_id": "doc9", "title": "Sistine Chapel - Renaissance Masterpiece", "text": "The Sistine Chapel is a chapel in the Apostolic Palace, the official residence of the Pope, in Vatican City. It is famous for its architecture and its decoration that was painted by various most famous artists of the High Renaissance, including Michelangelo.", "metadata": {}}
+{"_id": "doc10", "title": "Grand Canyon - Natural Wonder of the World", "text": "The Grand Canyon is a steep-sided canyon carved by the Colorado River in Arizona, United States. The Grand Canyon is 277 miles (446 km) long, up to 18 miles (29 km) wide and over a mile (1.6 km) deep.", "metadata": {}}
+{"_id": "doc11", "title": "Christopher Lloyd", "text": "Christopher Allen Lloyd (born October 22, 1938)[1] is an American actor, voice actor, and comedian. He is best known for his roles as Emmett \"Doc\" Brown in the Back to the Future trilogy, Judge Doom in Who Framed Roger Rabbit (1988), Merlock the Magician in DuckTales the Movie: Treasure of the Lost Lamp (1990), Uncle Fester in The Addams Family (1991) and its sequel Addams Family Values (1993), and Grigori Rasputin in Anastasia (1997).", "metadata": {}}
+{"_id": "doc12", "title": "Tobacco advertising", "text": "After World War II, cigarette companies advertised frequently on television programs. To combat this move by the cigarette companies, the Federal Communications Commission required television stations to air anti-smoking advertisements at no cost to the organizations providing such advertisements. In 1970, Congress took their anti-smoking initiative one step further and passed the Public Health Cigarette Smoking Act, banning the advertising of cigarettes on television and radio starting on January 2, 1971. After the television ban, most cigarette advertising took place in magazines, newspapers, and on billboards. However, in 1999 all cigarette billboard advertisements were replaced with anti-smoking messages, with some of these anti-smoking messages playing parodies of cigarette companies advertising figures and slogans. Since 1984, cigarette companies have also been forced to place Surgeon's General warnings on all cigarette packs and advertisements because of the passing of the Federal Cigarette Labeling and Advertising Act.[8] Restrictions on cigarette companies became even tighter in 2010 with the Family Smoking Prevention and Tobacco Control Act. The act prohibits tobacco companies from sponsoring sports, music, and other cultural events and also prevents the display of their logos or products on T-shirts, hats, or other apparel.[9] The constitutionality of both this act and the Food and Drug Administration's new graphic cigarette warning labels are being questioned under cigarette companies' first amendment rights.[10]", "metadata": {}}
+{"_id": "doc13", "title": "Beti Bachao, Beti Padhao Yojana", "text": "In 26 August 2016, Olympics 2016 bronze medallist Sakshi Malik was made brand ambassador for BBBP.[7]", "metadata": {}}
+{"_id": "doc14", "title": "List of Prison Break episodes", "text": "The first five seasons of Prison Break have been released on DVD and Blu-ray in Regions 1, 2, and 4. Each DVD boxed set includes all of the broadcast episodes from that season, the associated special episode, commentary from cast and crew, and profiles of various parts of Prison Break, such as Fox River State Penitentiary or the tattoo.[4][5][6][7] Prison Break is also available online, including iTunes, Amazon Video, and Netflix.[8][9] After the premiere of the second season of Prison Break, Fox began online streaming of the prior week's episode, though it originally restricted viewing to the United States.[10]", "metadata": {}}
+{"_id": "doc15", "title": "Atomic bombings of Hiroshima and Nagasaki", "text": "By August 1945, the Allies' Manhattan Project had produced two types of atomic bomb, and the 509th Composite Group of the United States Army Air Forces (USAAF) was equipped with the specialized Silverplate version of the Boeing B-29 Superfortress that could deliver them from Tinian in the Mariana Islands. Orders for atomic bombs to be used on four Japanese cities were issued on July 25. On August 6, one of its B-29s dropped a Little Boy uranium gun-type bomb on Hiroshima. Three days later, on August 9, a Fat Man plutonium implosion-type bomb was dropped by another B-29 on Nagasaki. The bombs immediately devastated their targets. Over the next two to four months, the acute effects of the atomic bombings killed 90,000\u2013146,000 people in Hiroshima and 39,000\u201380,000 people in Nagasaki; roughly half of the deaths in each city occurred on the first day. Large numbers of people continued to die from the effects of burns, radiation sickness, and other injuries, compounded by illness and malnutrition, for many months afterward. In both cities, most of the dead were civilians, although Hiroshima had a sizable military garrison.", "metadata": {}}
+{"_id": "doc16", "title": "Two-party system", "text": "There is general agreement that the United States has a two-party system; historically, there have been few instances in which third party candidates won an election. In the First Party System, only Alexander Hamilton's Federalist Party and Thomas Jefferson's Democratic-Republican Party were significant political parties. Toward the end of the First Party System, the Republicans dominated a one-party system (primarily under the Presidency of James Monroe). Under the Second Party System, the Democratic-Republican Party split during the election of 1824 into Adams' Men and Jackson's Men. In 1828, the modern Democratic Party formed in support of Andrew Jackson. The National Republicans were formed in support of John Quincy Adams. After the National Republicans collapsed, the Whig Party and the Free Soil Party quickly formed and collapsed. In 1854, the modern Republican Party formed from a loose coalition of former Whigs, Free Soilers and other anti-slavery activists. Abraham Lincoln became the first Republican president in 1860.", "metadata": {}}
+{"_id": "doc17", "title": "Equilateral triangle", "text": "An equilateral triangle is the most symmetrical triangle, having 3 lines of reflection and rotational symmetry of order 3 about its center. Its symmetry group is the dihedral group of order 6 D3.", "metadata": {}}
+{"_id": "doc18", "title": "The O.C. (season 4)", "text": "The fourth and final season of The O.C., an American teen drama television series, aired in the United States from November 2, 2006 to February 22, 2007 and consisted of sixteen episodes. The O.C's final season aired Thursdays at 9:00\u00a0p.m. ET in the United States on Fox, a terrestrial television network.[1] Fox tested a new timeslot by airing the second episode on Wednesday at 9:00\u00a0p.m. ET against ABC's Lost and CBS's Criminal Minds.[2] With Lost about to enter a three-month hiatus, Fox hoped that changing the timeslot for The O.C. would attract back viewers it had lost since the end of the previous season; however, the move was unsuccessful and the show returned to its Thursday timeslot.[3]", "metadata": {}}
+{"_id": "doc19", "title": "Crossing the Bar", "text": "The extended metaphor of \"crossing of bar\" represents traveling serenely and securely from life through death. The Pilot is a metaphor for God, whom the speaker hopes to meet face to face. Tennyson explained, \"The Pilot has been on board all the while, but in the dark I have not seen him\u00e2\u20ac\u00a6[He is] that Divine and Unseen Who is always guiding us.\"[1]", "metadata": {}}
+{"_id": "doc20", "title": "Protectionism", "text": "Protectionism is the economic policy of restricting imports from other countries through methods such as tariffs on imported goods, import quotas, and a variety of other government regulations. Proponents claim that protectionist policies shield the producers, businesses, and workers of the import-competing sector in the country from foreign competitors. However, they also reduce trade and adversely affect consumers in general (by raising the cost of imported goods), and harm the producers and workers in export sectors, both in the country implementing protectionist policies, and in the countries protected against.", "metadata": {}}