Image by Editor | Midjourney
Natural language processing (NLP) enables us to extract meaningful insights from unstructured text data like reviews, tweets, or articles. While Python is widely used for NLP, R also has a rich set of tools for text analysis and modeling. In this article, we walk through the NLP workflow in R — from text preprocessing to building sentiment analysis models.
Text Preprocessing
Preprocessing is an essential step in NLP that transforms raw text data into a clean and structured format suitable for analysis.
It includes several steps:
- Convert to lowercase: Changes all text to lowercase (e.g., “Hello” → “hello”).
- Remove Punctuation and Numbers: Eliminates noise from text.
- Remove numbers: Useful if numbers don’t carry meaning.
- Remove stopwords: Removes common words like “the”, “is”, etc.
- Strip whitespace: Cleans up unnecessary spaces and tabs.
- Tokenization: Splits text into words (tokens).
- Stemming: Reduces words to their root form (e.g., “playing” → “play”).
# Load necessary libraries
library(tm)
library(dplyr)
library(SnowballC)
library(tidytext)
# Sample corpus
corpus <- VCorpus(VectorSource(c(
"Natural Language Processing is a fascinating field of study.",
"Python is widely used in data science and machine learning.",
"R programming is powerful for statistical analysis.",
"Sentiment analysis can reveal people's emotions from text.",
"Exploratory data analysis is essential before building machine learning models.",
"Text mining involves extracting useful information from unstructured text data.",
"Deep learning models have revolutionized the field of NLP.",
"AI models like GPT-4 can generate human-like text responses.",
"Data preprocessing is an important step in machine learning pipelines.",
"Cleaning data involves removing errors and inconsistencies from raw data.",
"Tokenization is an important step in text analysis for breaking sentences into words."
)))
# Preprocessing pipeline
corpus <- tm_map(corpus, content_transformer(tolower)) # Lowercase corpus
corpus <- tm_map(corpus, removePunctuation) # Remove punctuation
corpus <- tm_map(corpus, removeNumbers) # Remove numbers
corpus <- tm_map(corpus, removeWords, stopwords("english")) # Remove stopwords
corpus <- tm_map(corpus, stripWhitespace) # Strip whitespace
corpus <- tm_map(corpus, stemDocument) # Stemming
# Convert corpus to a data frame
corpus_df <- data.frame(text = sapply(corpus, as.character), stringsAsFactors = FALSE)
# Tokenize the text
tokens <- corpus_df %>%
unnest_tokens(word, text)
# Print the tokenized words
print(tokens)
Output:
word
1...1 natur
1...2 languag
1...3 process
1...4 fascin
1...5 field
1...6 studi
2...7 python
2...8 wide
2...9 use
2...10 data
2...11 scienc
2...12 machin
2...13 learn
3...14 r
3...15 program
3...16 power
3...17 statist
3...18 analysi
4...19 sentiment
4...20 analysi
4...21 can
4...22 reveal
4...23 peopl
4...24 emot
4...25 text
5...26 exploratori
5...27 data
5...28 analysi
5...29 essenti
5...30 build
5...31 machin
5...32 learn
5...33 model
6...34 text
6...35 mine
6...36 involv
6...37 extract
6...38 use
6...39 inform
6...40 unstructur
6...41 text
6...42 data
7...43 deep
7...44 learn
7...45 model
7...46 revolution
7...47 field
7...48 nlp
8...49 ai
8...50 model
8...51 like
8...52 gpt
8...53 can
8...54 generat
8...55 humanlik
8...56 text
8...57 respons
9...58 data
9...59 preprocess
9...60 import
9...61 step
9...62 machin
9...63 learn
9...64 pipelin
10...65 clean
10...66 data
10...67 involv
10...68 remov
10...69 error
10...70 inconsist
10...71 raw
10...72 data
11...73 token
11...74 import
11...75 step
11...76 text
11...77 analysi
11...78 break
11...79 sentenc
11...80 word
Exploratory Text Analysis
Exploratory text analysis (ETA) is the process of analyzing text data to find patterns and insights before using more advanced techniques. It helps to better understand the content and structure of the text.
Word Frequency
This visualization shows the most common words in your dataset after cleaning and tokenization. It helps identify dominant themes or topics based on word usage frequency.
# Word frequency word_freq <- tokens %>% count(word, sort = TRUE) # Bar plot for top 10 words word_freq %>% top_n(10) %>% ggplot(aes(x = reorder(word, n), y = n)) + geom_col(fill = "steelblue") + coord_flip() + labs(title = "Top 10 Word Frequencies", x = NULL, y = "Frequency") </code></pre>
Word Cloud
A word cloud visually represents word frequency, where more frequent terms appear larger and bolder. It helps quickly identify the most common words in a dataset. This makes it easier to spot trends and themes.
# Create a word cloud with a custom color palette
wordcloud(tokens$word,
min.freq = 1,
scale = c(3, 0.5),
colors = brewer.pal(8, "Dark2"))
Text Vectorization
Text vectorization is the process of converting textual data into numerical form, so that algorithms can understand and process it.
There are several methods for vectorizing text, each with different strengths and weaknesses. Let’s explore the most common techniques used for text vectorization:
Bag-of-Words (BoW)
In the Bag-of-Words model, the text is represented as a matrix where each row is a document, and each column corresponds to a word from the vocabulary. The values in the matrix indicate the frequency of each word in the document.
# Create a Document-Term Matrix (DTM) for Bag-of-Words dtm <- DocumentTermMatrix(VCorpus(VectorSource(corpus_df$text))) # Convert DTM to a data frame dtm_df <- as.data.frame(as.matrix(dtm)) # Print the Document-Term Matrix print(dtm_df)
Output:
analysi break build can clean data deep emot error essenti exploratori extract 1 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 1 0 0 0 0 0 0 3 1 0 0 0 0 0 0 0 0 0 0 0 4 1 0 0 1 0 0 0 1 0 0 0 0 5 1 0 1 0 0 1 0 0 0 1 1 0 6 0 0 0 0 0 1 0 0 0 0 0 1 7 0 0 0 0 0 0 1 0 0 0 0 0 8 0 0 0 1 0 0 0 0 0 0 0 0 9 0 0 0 0 0 1 0 0 0 0 0 0 10 0 0 0 0 1 2 0 0 1 0 0 0 11 1 1 0 0 0 0 0 0 0 0 0 0 fascin field generat gpt humanlik import inconsist inform involv languag learn 1 1 1 0 0 0 0 0 0 0 1 0 2 0 0 0 0 0 0 0 0 0 0 1 3 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 1 6 0 0 0 0 0 0 0 1 1 0 0 7 0 1 0 0 0 0 0 0 0 0 1 8 0 0 1 1 1 0 0 0 0 0 0 9 0 0 0 0 0 1 0 0 0 0 1 10 0 0 0 0 0 0 1 0 1 0 0 11 0 0 0 0 0 1 0 0 0 0 0 like machin mine model natur nlp peopl pipelin power preprocess process program 1 0 0 0 0 1 0 0 0 0 0 1 0 2 0 1 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 1 0 0 1 4 0 0 0 0 0 0 1 0 0 0 0 0 5 0 1 0 1 0 0 0 0 0 0 0 0 6 0 0 1 0 0 0 0 0 0 0 0 0 7 0 0 0 1 0 1 0 0 0 0 0 0 8 1 0 0 1 0 0 0 0 0 0 0 0 9 0 1 0 0 0 0 0 1 0 1 0 0 10 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 0 0 0 python raw remov respons reveal revolution scienc sentenc sentiment statist step 1 0 0 0 0 0 0 0 0 0 0 0 2 1 0 0 0 0 0 1 0 0 0 0 3 0 0 0 0 0 0 0 0 0 1 0 4 0 0 0 0 1 0 0 0 1 0 0 5 0 0 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 1 0 0 0 0 0 8 0 0 0 1 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 0 1 10 0 1 1 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 1 0 0 1 studi text token unstructur use wide word 1 1 0 0 0 0 0 0 2 0 0 0 0 1 1 0 3 0 0 0 0 0 0 0 4 0 1 0 0 0 0 0 5 0 0 0 0 0 0 0 6 0 2 0 1 1 0 0 7 0 0 0 0 0 0 0 8 0 1 0 0 0 0 0 9 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 11 0 1 1 0 0 0 1
TF-IDF
TF-IDF helps find important words in a document by checking how often they appear, but it also reduces the importance of words that are common in many documents. It helps highlight unique and meaningful words for each text.
# Create a document-term matrix dtm <- DocumentTermMatrix(VCorpus(VectorSource(corpus_df$text))) # Convert DTM to matrix dtm_matrix <- as.matrix(dtm) # Perform TF-IDF transformation tfidf <- weightTfIdf(dtm) # Convert to a data frame for easier viewing tfidf_df <- as.data.frame(as.matrix(tfidf)) # View the TF-IDF matrix print(tfidf_df)
Output:
analysi break build can clean data deep emot
1 0.0000000 0.000000 0.000000 0.0000000 0.000000 0.0000000 0.0000000 0.0000000
2 0.0000000 0.000000 0.000000 0.0000000 0.000000 0.1625005 0.0000000 0.0000000
3 0.3648579 0.000000 0.000000 0.0000000 0.000000 0.0000000 0.0000000 0.0000000
4 0.2084902 0.000000 0.000000 0.3513474 0.000000 0.0000000 0.0000000 0.4942045
5 0.1824290 0.000000 0.432429 0.0000000 0.000000 0.1421879 0.0000000 0.0000000
6 0.0000000 0.000000 0.000000 0.0000000 0.000000 0.1263893 0.0000000 0.0000000
7 0.0000000 0.000000 0.000000 0.0000000 0.000000 0.0000000 0.5765719 0.0000000
8 0.0000000 0.000000 0.000000 0.3074290 0.000000 0.0000000 0.0000000 0.0000000
9 0.0000000 0.000000 0.000000 0.0000000 0.000000 0.1625005 0.0000000 0.0000000
10 0.0000000 0.000000 0.000000 0.0000000 0.432429 0.2843759 0.0000000 0.0000000
11 0.1824290 0.432429 0.000000 0.0000000 0.000000 0.0000000 0.0000000 0.0000000
error essenti exploratori extract fascin field generat gpt
1 0.000000 0.000000 0.000000 0.0000000 0.5765719 0.4099053 0.000000 0.000000
2 0.000000 0.000000 0.000000 0.0000000 0.0000000 0.0000000 0.000000 0.000000
3 0.000000 0.000000 0.000000 0.0000000 0.0000000 0.0000000 0.000000 0.000000
4 0.000000 0.000000 0.000000 0.0000000 0.0000000 0.0000000 0.000000 0.000000
5 0.000000 0.432429 0.432429 0.0000000 0.0000000 0.0000000 0.000000 0.000000
6 0.000000 0.000000 0.000000 0.3843813 0.0000000 0.0000000 0.000000 0.000000
7 0.000000 0.000000 0.000000 0.0000000 0.0000000 0.4099053 0.000000 0.000000
8 0.000000 0.000000 0.000000 0.0000000 0.0000000 0.0000000 0.432429 0.432429
9 0.000000 0.000000 0.000000 0.0000000 0.0000000 0.0000000 0.000000 0.000000
10 0.432429 0.000000 0.000000 0.0000000 0.0000000 0.0000000 0.000000 0.000000
11 0.000000 0.000000 0.000000 0.0000000 0.0000000 0.0000000 0.000000 0.000000
humanlik import inconsist inform involv languag learn like
1 0.000000 0.0000000 0.000000 0.0000000 0.0000000 0.5765719 0.0000000 0.000000
2 0.000000 0.0000000 0.000000 0.0000000 0.0000000 0.0000000 0.2084902 0.000000
3 0.000000 0.0000000 0.000000 0.0000000 0.0000000 0.0000000 0.0000000 0.000000
4 0.000000 0.0000000 0.000000 0.0000000 0.0000000 0.0000000 0.0000000 0.000000
5 0.000000 0.0000000 0.000000 0.0000000 0.0000000 0.0000000 0.1824290 0.000000
6 0.000000 0.0000000 0.000000 0.3843813 0.2732702 0.0000000 0.0000000 0.000000
7 0.000000 0.0000000 0.000000 0.0000000 0.0000000 0.0000000 0.2432386 0.000000
8 0.432429 0.0000000 0.000000 0.0000000 0.0000000 0.0000000 0.0000000 0.432429
9 0.000000 0.3513474 0.000000 0.0000000 0.0000000 0.0000000 0.2084902 0.000000
10 0.000000 0.0000000 0.432429 0.0000000 0.3074290 0.0000000 0.0000000 0.000000
11 0.000000 0.3074290 0.000000 0.0000000 0.0000000 0.0000000 0.0000000 0.000000
machin mine model natur nlp peopl pipelin power
1 0.0000000 0.0000000 0.0000000 0.5765719 0.0000000 0.0000000 0.0000000 0.0000000
2 0.2677813 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
3 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.8648579
4 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.4942045 0.0000000 0.0000000
5 0.2343086 0.0000000 0.2343086 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
6 0.0000000 0.3843813 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
7 0.0000000 0.0000000 0.3124115 0.0000000 0.5765719 0.0000000 0.0000000 0.0000000
8 0.0000000 0.0000000 0.2343086 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
9 0.2677813 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.4942045 0.0000000
10 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
11 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
preprocess process program python raw remov respons reveal
1 0.0000000 0.5765719 0.0000000 0.0000000 0.000000 0.000000 0.000000 0.0000000
2 0.0000000 0.0000000 0.0000000 0.4942045 0.000000 0.000000 0.000000 0.0000000
3 0.0000000 0.0000000 0.8648579 0.0000000 0.000000 0.000000 0.000000 0.0000000
4 0.0000000 0.0000000 0.0000000 0.0000000 0.000000 0.000000 0.000000 0.4942045
5 0.0000000 0.0000000 0.0000000 0.0000000 0.000000 0.000000 0.000000 0.0000000
6 0.0000000 0.0000000 0.0000000 0.0000000 0.000000 0.000000 0.000000 0.0000000
7 0.0000000 0.0000000 0.0000000 0.0000000 0.000000 0.000000 0.000000 0.0000000
8 0.0000000 0.0000000 0.0000000 0.0000000 0.000000 0.000000 0.432429 0.0000000
9 0.4942045 0.0000000 0.0000000 0.0000000 0.000000 0.000000 0.000000 0.0000000
10 0.0000000 0.0000000 0.0000000 0.0000000 0.432429 0.432429 0.000000 0.0000000
11 0.0000000 0.0000000 0.0000000 0.0000000 0.000000 0.000000 0.000000 0.0000000
revolution scienc sentenc sentiment statist step studi text
1 0.0000000 0.0000000 0.000000 0.0000000 0.0000000 0.0000000 0.5765719 0.0000000
2 0.0000000 0.4942045 0.000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
3 0.0000000 0.0000000 0.000000 0.0000000 0.8648579 0.0000000 0.0000000 0.0000000
4 0.0000000 0.0000000 0.000000 0.4942045 0.0000000 0.0000000 0.0000000 0.2084902
5 0.0000000 0.0000000 0.000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
6 0.0000000 0.0000000 0.000000 0.0000000 0.0000000 0.0000000 0.0000000 0.3243181
7 0.5765719 0.0000000 0.000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
8 0.0000000 0.0000000 0.000000 0.0000000 0.0000000 0.0000000 0.0000000 0.1824290
9 0.0000000 0.0000000 0.000000 0.0000000 0.0000000 0.3513474 0.0000000 0.0000000
10 0.0000000 0.0000000 0.000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
11 0.0000000 0.0000000 0.432429 0.0000000 0.0000000 0.3074290 0.0000000 0.1824290
token unstructur use wide word
1 0.000000 0.0000000 0.0000000 0.0000000 0.000000
2 0.000000 0.0000000 0.3513474 0.4942045 0.000000
3 0.000000 0.0000000 0.0000000 0.0000000 0.000000
4 0.000000 0.0000000 0.0000000 0.0000000 0.000000
5 0.000000 0.0000000 0.0000000 0.0000000 0.000000
6 0.000000 0.3843813 0.2732702 0.0000000 0.000000
7 0.000000 0.0000000 0.0000000 0.0000000 0.000000
8 0.000000 0.0000000 0.0000000 0.0000000 0.000000
9 0.000000 0.0000000 0.0000000 0.0000000 0.000000
10 0.000000 0.0000000 0.0000000 0.0000000 0.000000
11 0.432429 0.0000000 0.0000000 0.0000000 0.432429
Sentiment Analysis
Sentiment Analysis is a technique in NLP. It identifies and extracts emotions or opinions from text. The goal is to determine whether the sentiment is positive, negative, or neutral.
R provides several packages for sentiment analysis:
Using syuzhet
The syuzhet package in R is used for sentiment analysis of text using lexicons like NRC, Bing, and AFINN. It helps extract emotional values such as joy, anger, and trust from sentences.
# Sentiment using syuzhet library(syuzhet) cleaned_text <- corpus_df$text sentiments <- get_nrc_sentiment(cleaned_text) head(sentiments)
Output:
anger anticipation disgust fear joy sadness surprise trust negative positive 1 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 1 3 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 3 6 0 0 0 0 0 0 0 1 0 0
Using sentimentr
The sentimentr package in R analyzes sentiment by considering valence shifters such as negators, amplifiers, and de-amplifiers. This results in more accurate and nuanced sentiment scores at the sentence or document level.
# Sentiment using sentimentr library(sentimentr) results <- sentiment(cleaned_text) summary(results)
Output:
element_id sentence_id word_count sentiment Min. : 1.0 Min. :1 Min. :5.000 Min. :-0.08839 1st Qu.: 3.5 1st Qu.:1 1st Qu.:6.500 1st Qu.: 0.00000 Median : 6.0 Median :1 Median :7.000 Median : 0.00000 Mean : 6.0 Mean :1 Mean :7.273 Mean : 0.18813 3rd Qu.: 8.5 3rd Qu.:1 3rd Qu.:8.000 3rd Qu.: 0.30237 Max. :11.0 Max. :1 Max. :9.000 Max. : 0.88182
Using tidytext
The tidytext package in R can be used to find emotions or opinions in text by matching words with sentiment lexicons like AFINN or NRC. It breaks text into individual words and lets you calculate sentiment scores.
# Sentiment using tidytext library(tidytext) bing % inner_join(bing, by = "word") %>% count(sentiment, sort = TRUE)
Output:
sentiment n 1 negative 2 2 positive 2
Wrapping Up
Text preprocessing cleans and structures raw text for analysis. It includes lowercasing, removing punctuation and stopwords, tokenization, and stemming. Exploratory text analysis helps us understand the text. Word frequency analysis highlights common terms, and a word cloud visualizes frequent words. Text vectorization is another key step, where text is converted into numerical format using methods like Bag of Words or TF-IDF. Sentiment analysis detects emotions in text. Tools like syuzhet, sentimentr, and tidytext assess whether the text is positive, negative, or neutral.
