Information technology

In this project, you will use computational thinking to develop and then implement an algorithm to solve the problem of counting the number of occurrences of a word and its synonyms in a corpus of text documents.

This project consists of four parts; you will complete one part at the end of each course module:

Apply the four pillars of CT and describe the results of each
Express the algorithm used in the solution using a flowchart
Express the algorithm using a structured notation known as pseudocode
Implement the solution in Python
Description
less
With the dawn of the Information Age, the amount of data that is available on the World Wide Web has grown at incredible rates in recent years, and the ability to extract useful knowledge from that data — whether it’s for personal, social, or business reasons — is a problem that can be addressed using computational thinking.

In many cases, we start with individual documents — emails, social media posts, product reviews, etc. — and collect them into a corpus, and then want to search the documents in the corpus for a particular keyword, or search term, and its synonyms, which are words that have the same or similar meanings.

For example:

You may want to look at the words you are using in your own social media content (e.g., Facebook posts, Tweets, etc.) in order to get an understanding of your own wellbeing and mental health
You might want to analyze your emails to determine which topics you are frequently discussing
You may want to look at reviews written by other people, such as customer feedback on products your company produces, to get an idea of whether the general sentiment is positive or negative
Medical practitioners might want to see what words are being used in social media content to understand the spread of a disease
Researchers in linguistics may want to understand how the use of a particular word or phrase evolves over time
Although the outcomes of all of the scenarios will be slightly different, all of them involve a common element of determining how many times a word and its synonyms appear in some collection of documents.

Setup
less
There are three inputs to this problem:

Keyword: The word for which you want to conduct the search
Thesaurus: A set of words, each of which has associated synonyms
Corpus: A set of documents, each of which contains some number of words
The output of the solution to this problem should be the number of occurrences of the keyword and its synonyms in all the documents in the corpus.

For simplicity, in addressing this problem you do not need to worry about things like capitalization, partial word matching, punctuation, etc. and you do not need to worry about alternate spellings or word variations.

Step-By-Step Assignment Instructions
less
In this part of the project, you will apply the four pillars of CT to this problem by answering the following questions:

Using decomposition, what are the primary sub-problems that need to be solved in solving the overall problem?
Using pattern recognition, what patterns do you see in the solution, i.e., what processes need to be repeated?
Using data abstraction and representation, how would you represent the thesaurus, the corpus, and each of the documents in the corpus?
Using the results of the first three pillars, what is the algorithm that you would use to solve this problem? Describe it in as much detail as possible.
After answering those four questions, answer the following question as well:

5. Describe a problem that you may face — either in your career or in everyday life — that involves determining the number of occurrences of a word and its synonyms in a corpus of documents. The problem you face may be much bigger than that and require that calculation as only a small part of the solution, but should involve looking through some collection of text and looking for certain words.

Regardless of your answer for question #5, be sure that your answers for questions #1-4 address the general problem of counting the number of occurrences of a words and its synonyms in a corpus of text documents, and not the specific problem you described in #5.

Hints

Keep in mind that your answers for questions #1-4 should only focus on the general problem of counting the number of occurrences of a words and its synonyms in a corpus of text documents.
For decomposition, you should be able to decompose this problem into two sub-problems, one of which needs to be done before the next.
For pattern recognition, you should be able to identify at least two patterns. Think about which inputs to the problem are collections and what patterns exist that can be applied to each element of the collections.
For data representation and abstraction, think about relationships between the inputs, how each is composed (e.g., whether something is a collection), and the minimum amount of information you need for each.
For the algorithm, use the results of the decomposition pillar to think about the smaller parts of the algorithm, and the result of the pattern recognition pillar to think about what you need to do repeatedly within each part.