Code for downstream task evaluation

README.md

@@ -235,3 +235,39 @@ This command will first build the BM25 index over the training data, run search
 
 There are a few hyperparameters you can explore, e.g., `--ensemble_k`, which uses the top k documents instead of top 1 and use them in parallel through an ensembling approach, following the REPLUG recipe. We find this significantly increases run-time while marginally improving PPL, so recommend not specifying it for your first run.
 
+
+### Downstream tasks evals
+We currently support these downstream tasks: `agn, amazon, cr, dbpedia, hyp, mr, rotten_tomatoes, rte, sst2, yelp`.
+
+#### Process IMDB data 
+
+Donwload IMDB corpus using the following the command. 
+```
+python scripts/download_data.py --subset imdb --split train
+```
+Then you can follow the above instruction to encode and build index of the corpus. 
+
+#### Run downstream tasks evaluation
+For downstream tasks, you need to specify the datastore you want to use (raw_file, index_path). There are several parameters you can explore including `K` (number of retrieved tokens), `KNN_TEMP` (temperature for kNN distribution), `inter_lambda` (interpolation lambda). 
+Here is one example to run evaluation on `cr` using `amazon` corpus. 
+
+```bash
+raw_file="new-amazon" # new-amazon # cc-news # Wikipedia_(en)
+model=kernelmachine/silo-pdsw-1.3b
+dataset=cr
+K=1024
+KNN_TEMP=1
+inter_lambda=0.7
+
+PYTHONPATH=. python scripts/eval.py \
+--model ${model}  \
+--knn_model ${model} \
+--n_sample 3000 \
+--raw_file ${raw_file} \
+--inter_lambda $inter_lambda \
+--dataset_dir data_eval/benchmark/$dataset \
+--k $K \
+--dataset_name $dataset \
+--batch_size 5 \
+--knn_temp ${KNN_TEMP} 
+```

data_eval/fuzzy_verbalizer/agn_verbalizer.txt

@@ -0,0 +1,4 @@
+politics, government, diplomatic, law, aristotle, diplomatical, governance, republic, politician, smooth, suave, state, expedient, sagacious, police, election, political, monarchy, parliament, dukes, polity, regime, democratic, ethics, communism, federation, anarchism, authoritarianism, populism, bland, aristocracy, tribe, power, negotiation, force, warfare, city, clans, tribes, company, country, plato, confucius, latin, polis, kingship, earls, counts, tribute, lordship, property, inheritance, confiscation, individualist, allegiance, espionage, conspiracy, treason, jewish, gentile, convention, observance, celibacy, pope, taxation, petition, legislation, majority, collegial, permeates, flabby, policy, governmental, pervades, abstract, straitjacket, justice, myopic, discernment, curfew, consciences, revenue, pervade, matriarchal, numbed, juridical, rightness, unaccountable, clannish, deliberative, monopoly, fatness, paternalism, monkish, coin, principle, sinfulness, morass, permeate, stuffy, timorous, meddles, factious, disentangle, grayness, etheric, blandness, indigent, expediency, comity, unclothed, subjectivity, transitory, guild, cesspool, psyche, quicksand, egoism, diplomat, colonies, agreement, profit, policy-making, education, democracy, debate, anarchy, executive, humorless, colorblind, monarchies, psychodrama, self-perpetuating, piercer, clubby, reflation, nontransparent, kafkaesque, postindustrial, democracies, unsustainability, socialism, oligarchies, self-regulating, hidebound, nonideological, wrongness, tyrannies, dispassion, clinton, governor, senate, squishy, soviet, heteronormative, administration, corporatism, metabolizes, covenantal, microbiota, shapers, solipsistic, originalism, seven-man, empowerment, standard-setting, groupthink, bureaucracy, sovereignty, 146-nation, autocracy, constitutions, president, imperialism, privatization, restoration, presidential, preside, manifesto, interpol, france, minimalist, crafty, mastermind, federal, racketeer, campaigner, australia, contract, capitalism, administrative, presidency, ombudsman, confederation, generalissimo, fiefdom, officer, senator, ceo, autocratic, financier, secede, timocracy, anarchist, manipulative, civilization, civil, artifice, bloc, suzerainty, politicize, issue, multinational, shrewd, politricks, liberalism, warden, privatize, dishonest, federalization, govern, tenderpreneur, treasurer, authority, international, separatism, expert, statesperson, economist, chancellor, commissioner, guile, neocolonialism, impolitic, politik, papacy, socio, posturing, discourse, politico, chomsky, democrats, crist, polemics, hegelian, religio, politicians, elites, quietism, theo, circumlocution, parliamentary, sayers, roguery, sociopolitical, bonapartist, moralize, reactionary, religionist, egotistic, pragmatics, contestation, apolitical, raison, poli, pols, intelligentsia, viler, hausa, jurisdiction, manipulation, cabal, resourceful, democrat, judiciary, supremacy, demagogue, controller, shrewdness, corporatist, official, socialist, wizard, dominion, supervision, skillful, organisation, cleverness, statesman, employer, undeceive, comptroller, organization, magician, mislead, rebellion, civility, reich, marxism, cosmopolitanism, fedzilla, neoimperialism, leader, quango, corruption, extortion, misleader, nepotism, patronage, crossbencher, graft, mandarinate, scienda, stateswoman, embezzlement, presider, statocracy, politick, policial, ocracy, reformable, exclusionism, egoistical, coopt, laic, bureaucratize, liberalist, reactionism, confederal, popularism, delegitimation, realpolitik, establishmentarian, plebiscitary, confessionalism, demagogical, duumvirate, ethnocentric, coarsen, centrism, peoplehood, ideologic, liberalness, repub, revanchist, technocracy, balkanise, conscionable, christianism, contradictive, mobocracy, equalitarian, sermonise, mudslinger, sloganeer, technocratic, biopolitics, megalomanic, westernism, passivism, russophile, falsifiable, u.s.s.r., presidium, nativism, elections, misinform, nationalise, ideology, constitutionalization, disenchant, coalition, cronyism, influence, hobbes, kleptocracy
+sports, athletics, gymnastics, sportsman, competition, cycling, soccer, tennis, game, downfield, offside, judo, polo, team, skiing, hockey, baseball, football, fun, sportswoman, play, rugby, basketball, call, spar, kill, referee, ineligible, wipeout, schuss, luge, archery, upfield, funambulism, toboggan, skateboard, jackknife, ski, rollerblade, boast, mutation, lark, frolic, frisk, skylark, gambol, feature, disport, mutant, romp, cavort, rollick, coach, volleyball, athletic, sumo, television, sportsmanship, pastime, badminton, chess, position, sportaccord, equestrianism, sportsperson, athlete, competitions, golf, coaches, wrestling, cricket, championships, racers, challenge, motocross, leagues, variation, boxing, summercater, entertainment, tournament, champion, season, playoffs, athleticism, dexterity, foul, jog, handler, box, defense, defence, trial, series, cut, side, english, tuck, canoe, dribble, possession, bob, stroke, shot, equitation, row, aquatics, recreation, diversion, toss, pass, flip, occupation, line, job, paddle, carry, surf, racket, drive, surfboard, punt, onside, kick, submarine, bandy, kayak, drop, pack, umpire, backpack, scull, snorkel, shoot, rappel, field, mountaineer, start, curl, underarm, seed, surge, turn, underhand, underhanded, round, bout, hurdle, average, sleigh, loose, overhand, humor, legal, humour, wit, lead, hike, deficit, timer, witticism, jocularity, timekeeper, shooter, scout, home, ref, ironman, skate, manager, comedy, sportive, lacrosse, tradition, goal, biathlon, dodgeball, leisure, floorball, soccerplex, overarm, waggishness, jocosity, wittiness, windsurf, abseil, skin-dive, double-team, prizefight, outclass, shadowbox, birling, spread-eagle, offsides, man-to-man, one-on-one, most-valuable, waggery, motorsport, sportful, sporter, gameday, sportsaholic, nonsports, multisport, footballer, outsport, sportless, lusorious, acrobatic, sportlike, rugger, paddlesport, sportsplex, gamesome, pickleball, postseason, professional, passtime, competitive, slalom, birle, skateboarder, olympics, world, racquet, compete, bowling, competes, olympic, dropkick, sportsfield, clubs, skater, formula, racer, cheerlead, pharaoh, race, minigame, athletes, recreational, bike, snowboard, bicycle, championship, motorcycle, brand, youth, nascar, iran, model, f1, uci, teams, puck, track, racquetball, competitor, riders, postgame, subbuteo, enthusiasts, trashsport, popular, super, games, jousting, class, sponsorship, event, netball, softball, models, best, women, amateur, association, experience, peloponnese, car, venue, players, roller, fia, pigskin, fit, standards, drivers, european, national, tour, fitness, cars, esports, transgender, wogball, bucketball, tennikoit, snowsport, nongame, gamification, subgame, nongamer, vacationer, gameplayer, rioting, gaymer, sportsbook, hooliganism, zourkhaneh, gameography, watersport, fanwear, cross-country, soccerball, wintersports, woodball, concussion, disability, motorsports, interscholastic, tournaments, korfball, triathlon, intercollegiate, paralympic, olympian, bullfighting, boxers, subculture, crosscountry, mma, equestrian, wage, salary, indoors, pay-per-view, fina, nonresident, spectatorship, waterpolo, bloodsport, bobsledding, tourn, riflery, alpinism, bundesliga, nfl, fanatic
+business, commerce, trade, market, retail, traffic, commercial, marketing, exchange, sell, deal, export, shop, transportation, finance, barter, noncommercial, resell, mercantilism, arbitrage, commercialism, commercialize, wholesale, negociate, transaction, merchandise, transact, auctioneer, doc, smuggle, import, deaccession, importation, merchant, merchandising, mercantile, trading, sale, exportation, e-commerce, affairs, goods, commodity, agriculture, resources, tourism, telecommunications, economic, consumer, agricultural, banking, communications, industries, globalization, trader, consumers, seller, resale, vendor, huckster, shipping, broker, pawn, ipo, hock, interchange, transport, evasion, distribution, antique, purchase, conversation, franchise, noaa, clear, auction, browse, trust, soak, stock, remainder, payment, dealings, realise, prehistoric, retailer, peddle, basketry, tradesman, vendible, simony, mercature, tradeful, untradeable, tradesfolk, tradable, swap, mercat, dicker, untraded, mart, oversold, dealer, transactive, intertraffic, dressmaking, marketable, hanse, venal, monger, vend, hoppo, troak, marketplace, cybercommerce, scorse, defrayment, liquidize, comparison-shop, impulse-buy, commercialise, defrayal, nonpayment, usance, sellable, overtrade, protrade, incoterm, faculty, bureau, self-sufficiency, nonexchange, pawnbrokery, exchangeable, chaffer, downtick, postdeal, department, cheap, tradesperson, mastercraftsman, carpentry, commission, cooperage, undealt, dealy, reexchange, vice, intercourse, investment, furriery, buyer, haggle, sector, merchants, sectors, forestry, office, contraband, issues, general, nonmarket, bootleg, labor, marketeer, misdeal, marketwide, enterprises, committee, china, kong, hong, local, higgler, circa, services, planning, secretary, taiwan, oversees, shopping, markets, employment, today, public, ministry, press, foreign, chairman, forum, rebuy, domestic, firm, butcherdom, board, regional, report, economics, environment, immigration, financial, institutional, firms, exchanges, customs, petroleum, corporations, xinhua, wto, currency, cornmarket, brokering, logroll, marketman, undersell, trades, handel, store, excambion, peddlery, oligopoly, cheesemonger, act, barterer, money, cybermarket, moc, woolhall, signatures, tradecraft, comercio, commercio, entrepreneurship, retailing, maritime, businesses, chamber, marketplaces, arts, marts, transactions, textiles, humanities, exports, lifeblood, prosperity, portals, outbound, remittance, agri, bookselling, cyberspace, telecom, steamboats, cultural, facilitation, businesspeople, industrie, protectionism, gateway, unctad, floriculture, connectivity, closeout, métier, truckman, oversell, craftsmaster, botanica, redeal, monopsony, buyback, patela, numismatist, barkeeping, ndrc, u.s., commerical, usdoc, e-business, entrepot, containerization, busi, vitalization, agrarianism, capital, corporation
+technology, engineering, science, biotechnology, internet, nanotechnology, robotics, communication, computer, industry, automation, wheel, technological, equipment, manufacturing, application, bionics, energy, technical, ergonomics, scientific, telephone, development, dolphin, systems, software, transhumanism, electronics, digital, tech, devices, tools, applications, computers, capabilities, expertise, biomedical, innovations, wireless, products, solutions, management, language, engineer, homo, neolithic, bioscience, scientist, biology, skill, good, service, knowledge, prehistory, lever, weapon, club, miniaturization, economy, pollution, value, productivity, discipline, bailiwick, machinery, source, bioengineering, subject, rocketry, sink, study, primates, technologies, crowbar, spoon, neo-luddism, anarcho-primitivism, ee, medicine, techno-progressivism, physics, innovation, phenomenal, tool, formality, developed, research, utility, merriam-webster, usability, electronic, safety, micro, focus, mathematics, advanced, multimedia, history, system, networking, electron, semiconductor, innovative, enterprise, global, hardware, uses, core, product, data, sophisticated, industrial, design, state-of-the-art, makers, components, use, processing, hominids, cyberculture, create, information, mobile, bipedal, companies, programs, creative, expand, strategy, new, program, generation, labs, dynamic, aims, networks, enables, component, capability, build, specialized, wood, charcoal, clothing, genome, eurasia, deforestation, goal-oriented, technician, nomad, uruk, sumer, hierarchy, pseudoscience, irrigation, neuroscience, furnace, bellows, forge, gold, metrology, copper, geology, silver, evolution, alloys, bronze, brass, alchemy, steel, cyberscience, climatology, sociology, evolve, biophysics, iraq, bionanoscience, alchemical, evolutionary, tribology, sedentism, silk, transformation, systematics, phrenology, superscience, technoscience, optimization, transmutation, horseshoe, microscopy, multiscience, developer, construction, antiscience, geophysics, proscience, transformational, demography, psychology, organon, conversion, scienceless, screw, ic, chasten, geoscience, converter, metamorphosis, pulley, fortran, cybernetics, nonscience, transform, wheelbarrow, architectonics, transformer, sciencelike, windmill, radiography, transmute, agronomy, clock, glycoscience, mutate, convert, technique, complicate, hydroscience, mcscience, theoretician, genetics, environmental, physic, assimilate, metamorphic, mining, physiology, metallurgy, interoperable, liberalize, innovate, electrochemistry, biologist, change, biological, ecological, deaden, reform, metaphysics, electricity, agrobiology, photoscience, decarboxylate, flight, complexify, ology, chemistry, acetylate, volatilize, skyscraper, transaminate, motor, logy, telegraph, biometrics, technologists, microelectronics, innovators, holography, algorithms, informatics, diagnostics, pbc, nano, inventions, sapir, startups, methodologies, telematics, functionality, gadgets, ione, biomedicine, interfaces, prognostics, semiconductors, cryptography, ignis, geospatial, cryogenics, radio, lifehack, airplane, techie, sociobiology, automobile, saponify, biocomputing, rarefy, actinochemistry, technoid, geroscience, exobiology, convertee, transmogrify, opacify, conversive, transchange, neurophysics, electrotelegraphy, classicize, transistor, hydrolyze, downshift, cyberpsychology, professionalize, microphonics, unscramble, remew, inactivate, conventionalize, bioclimatology, sysop, brutalize, satellite, telecommunication, miniaturisation, technologic, tecnology, gizmo, photomicrography, energid, fiberoptics, mirasol, telerobotics, relume, fluidics, idesia, blueshift, lightwave, ceroma, corrigent, reflectent, vocable, seawell, photomultipliers, echoscope, electromagnetics, ultrasonics, viridity, micrographics, orthogon, chiliad, autonomics, aits, tomograph, biomimetics, vadium, snocone, pyrometers, architecture, faust, goethe, technicism, citizenship