Search Engine Competition Assignment
Uploaded by
Axellent4 AxellentSearch Engine Competition Assignment
Uploaded by
Axellent4 AxellentCommon questions
Powered by AIWhen running the competition script, the expected outputs include precision at 10 documents and the Mean Average Precision (MAP) for the training queries. These metrics help quantify the effectiveness of the search engine. The competition evaluates success by comparing the MAP values obtained from training queries to those from testing queries, aiming for a close match indicating robust search engine performance .
To set up and run the assignment, first download and extract Assignment_2.tar.gz into the parent directory of MeTA, and then use the terminal to change to the Assignment_2 directory. Execute the setup.sh script to move necessary data. Recompile MeTA by navigating to Assignment_2/build and running cmake and make commands. To run the competition program, make sure the config.toml points to the new ranker and compile MeTA again. Ensure to submit the results using python grader.py, creating a submit.txt file which should be uploaded under the assignment's My Submission tab .
Query expansion using synonyms can enhance search results by capturing a broader linguistic scope that might match more relevant documents that use different terminology. This can be particularly beneficial in retrieving documents with semantically similar content that might not share exact query terms. However, care must be taken to avoid topic drift, often addressed by giving original query terms more weight compared to the added synonyms. This technique can increase the recall of relevant documents while maintaining precision .
The training data file containing 100 queries with relevance judgments is used to optimize the search engine's parameters and retrieval methods by adjusting them to maximize metrics like precision and MAP. For effective optimization, one should ensure that the training data performance closely predicts testing outcomes. The testing data file contains 538 queries for final assessment, making it crucial to verify that optimizations transfer to unseen data, maintaining the robustness of the search engine .
The configuration file config.toml can be adjusted by changing the ranker parameter to use different built-in ranking functions, such as switching from BM25 to Jelinek-Mercer smoothing and tuning its lambda parameter. Additionally, the analyzers tag can be modified to experiment with different tokenization parameters and text filters, optimizing for the combination that yields the best performance. Altering the stopwords list can also potentially enhance performance by reducing noise from irrelevant terms .
The assignment suggests experimenting with different retrieval functions by implementing new ranking functions or tuning existing ones. For example, one could combine scores from multiple retrieval formulas, like BM25 and Dirichlet Prior, using a weighted approach to generate a composite score. The assignment provides a scaffold in competition.cpp to implement this kind of function, where experimentation with parameter tuning is also encouraged. Using non-programming techniques like adapting built-in rankers and varying tokenization and filtering parameters are additional suggested strategies .
Rocchio's method can be implemented by writing a new function in competition.cpp that modifies a query using positive feedback documents. The function takes a query and a set of top retrieved positive feedback documents, processes them, and outputs a modified query. This involves calling the Rocchio function on each query with its top 10 initially retrieved documents, then re-running the scoring function on the modified query. This process is incorporated in both the training and testing phases, replacing the original ranking score line with a new one to utilize the modified query for improved retrieval performance .
Non-programmatic methods to enhance retrieval performance include experimenting with different built-in rankers, such as the Jelinek-Mercer smoothing function, and tuning their parameters. Additionally, users are encouraged to test various tokenization strategies and text filters within the config.toml file, and modify the default stopwords list to include or exclude terms that might impact retrieval performance. Reindexing the dataset after such changes is essential for these modifications to take effect .
The automated grader evaluates the effectiveness of the search engine by calculating performance metrics such as precision and MAP from the submitted output file (submit.txt). It provides feedback on the performance, guiding students in improving their search engine's retrieval capabilities. Moreover, the grader handles submission validation, such as ensuring unique nicknames for submissions .
The dual-loop approach in competition.cpp serves two main purposes. The first loop iterates over 100 training queries to calculate and print metrics like precision at 10 documents and MAP, fostering search engine optimization during development. The second loop processes 538 testing queries, extracting top document IDs for output, providing a comparative evaluation of the search engine's performance against unseen data. This separation ensures robust evaluation and optimization .