top of page
Search

Measuring Controversy in Social Networks through NLP

Social media platforms have become a central hub for discussions on a wide range of topics, including those that are highly controversial. These platforms provide users with the ability to express their opinions freely, leading to vibrant but often polarized debates. The nature of these discussions can be both enriching and divisive, depending on the topic at hand.


Existing literature highlights several significant issues that arise from controversy in social networks. One major issue is the splitting of communities. When users engage in debates on controversial topics, they often form subgroups with like-minded individuals, leading to echo chambers where only similar viewpoints are shared and reinforced. This phenomenon can deepen divisions within the larger community, making it difficult for diverse perspectives to coexist.



Artistic representation of the polarization of social networks.


Another problem is the spread of biased information. In the heat of controversial discussions, information is frequently shared without proper verification. This can lead to the propagation of misinformation and disinformation, which in turn can skew public perception and understanding of the issues being debated. The presence of biased information further entrenches polarized views and undermines constructive dialogue.


Furthermore, controversial topics often give rise to hateful discussions and attacks between groups. The anonymity provided by social media platforms sometimes emboldens users to engage in hostile and aggressive behavior. Such interactions can escalate quickly, resulting in personal attacks, harassment, and even coordinated hate campaigns. These toxic environments not only affect the individuals involved but also contribute to a broader culture of intolerance and hostility online.


Why?


Identifying controversy in discussions on social networks is not only crucial but also opens the door to applying various strategies aimed at mitigating its adverse effects. Effective identification allows platforms to take proactive measures to foster healthier and more constructive interactions. Some key strategies include:


  1. Improving the “news diet”: By identifying controversial discussions, platforms can diversify the content that users are exposed to. This involves curating a balanced mix of news and perspectives, which can help break the cycle of echo chambers. A more varied “news diet” exposes users to differing viewpoints, fostering a more well-rounded understanding of contentious issues.

  2. Bridging echo chambers: Echo chambers are environments where users only encounter information and opinions that reinforce their preexisting beliefs. By detecting areas of controversy, interventions can be designed to connect users from different echo chambers. This can be done through promoting discussions across diverse groups, recommending content from varied sources, and encouraging respectful dialogue between opposing viewpoints. Bridging these gaps is essential for reducing polarization and promoting a more cohesive community.

  3. Defending from attacks: Controversial topics often attract negative behaviors such as trolling, harassment, and coordinated attacks. Early detection of such discussions allows for the implementation of defense mechanisms. These can include automated moderation tools to filter out abusive content, providing users with resources to protect themselves from online harassment, and implementing stricter community guidelines to deter hostile behavior. Defending users from attacks helps create a safer and more inclusive online environment.

In this article, we explain a new vocabulary-based technique[1] for identifying and quantifying controversy in discussions. This technique leverages Natural Language Processing (NLP) to analyze the language used in social media posts and detect signs of controversy. By focusing on specific vocabulary and linguistic patterns associated with contentious debates, their method aims to provide a more accurate and nuanced understanding of the levels of controversy present in online discussions. This approach not only helps in pinpointing controversial topics but also in measuring the intensity of the controversy, enabling more targeted and effective mitigation strategies.


Related Work


A significant body of research has been dedicated to quantifying the polarization observed in online social networks. Understanding and measuring polarization is essential for addressing the negative consequences of controversial discussions.

The primary characteristic of most of these previous works is their reliance on graph-based measures. These approaches typically involve constructing graphs where nodes represent users or pieces of content, and edges represent interactions such as likes, comments, or shares. By analyzing the structure and properties of these graphs, researchers can identify clusters of users with similar viewpoints and measure the extent of polarization within the network.


One notable contribution in this area is by Garimella et al. Their work, titled "Quantifying Controversy on Social Media"[2] provides an extensive comparison of various controversial measures. They explore different graph-building approaches and utilize a wide range of data sources to assess the effectiveness of these measures. Garimella et al. achieved the best performance among the methods evaluated, highlighting the robustness and accuracy of their approach.


In their study, Garimella et al. emphasize the importance of choosing appropriate graph-building techniques to accurately reflect the underlying social dynamics. They also demonstrate how different data sources can impact the measurement of controversy, underscoring the need for comprehensive data collection and analysis.

While graph-based methods have proven effective, they are not without limitations. These approaches often require substantial computational resources and can be sensitive to the quality and completeness of the underlying data. Additionally, they may not capture the full nuance of language and context in controversial discussions.

In response to these challenges, we discuss a new method for identifying and quantifying controversy. This method leverages the linguistic features of social media posts to detect signs of controversy, providing a complementary approach to existing graph-based methods. By focusing on the specific vocabulary and linguistic patterns associated with contentious debates, we aim to offer a more scalable and context-sensitive solution for measuring controversy in online discussions.


Datasets


In this study, the authors utilized 30 different discussions that occurred between 2015 and 2020, with an even split between those that involved controversy and those that did not. These discussions spanned six languages: English, Portuguese, Spanish, French, Korean, and Arabic.


Given that their models require a substantial amount of text for effective training and considering that a tweet contains no more than 240 characters, they set a minimum threshold of at least 100,000 tweets per discussion. This ensures that they have a sufficient volume of data to accurately train and evaluate their models.


To select new discussions and determine their level of controversy, they adopted the following criteria:


  1. Controversial Discussions:

  • They focused on topics that were widely covered by mainstream media and generated significant online and offline discussions.

  • Examples include political events such as elections, corruption cases, or judicial decisions.

  • To validate their selection, they manually reviewed a sample of tweets to confirm the presence of controversy.

  1. Non-Controversial Discussions:

  • They concentrated on “soft news” topics such as entertainment, impactful events, and dramatic stories that are less likely to polarize opinions.

  • These topics are typically more neutral and tend to foster less divisive discussions.

To further establish the presence or absence of controversy in their datasets, they employed network visualization techniques using ForceAtlas2[3]. This method allowed them to visualize the structure and dynamics of the discussions, helping to identify clusters and patterns indicative of controversy or the lack thereof. In the following picture, we can see how ForceAtlas2 plots graphs with controversy and without.



ForceAtlas2 layout over 6 different discussions on Twitter. The top images are highly polarized while the bottom ones are not.
ForceAtlas2 layout over 6 different discussions on Twitter. The top images are highly polarized while the bottom ones are not.

By carefully selecting and validating their datasets, they ensured a robust foundation for their vocabulary-based technique to identify and quantify controversy in social network discussions. In the next table, we can see a summary of some of the discussions used.


Hashtag/Key word

Number of Tweets

Description and period

LeadersDebate

250 000

Candidates debate, Nov 11-21,2019

Pelosi

252 000

Trump Impeachment, Dec 06,2019

@mauriciomacri

309 603

Mentions to Argentinian ex-president, Apr 05-11, 2018

Kavanaugh

260 000

Kavanaugh nomination, EEUU, Oct 05, 2018

Bolsonaro

170 764

Brazilian elections, Oct 27, 2018

Al-HilalEntertainment

221 925

Al-Hilal champion, Dec 01,2019

MiracleOfChristmasEve

251 974

Segun Woo singer birthday, 23-12-2019

Notredam

200 000 

Notredam fire, Apr 16, 2019

Wrestlemania

260 979

Wrestlemania event, Apr 08,2019

Messi

200 000

Lionel Messi Birthday, Jun 24, 2019


Methodology


The researchers propose a novel technique for quantifying controversy through a vocabulary-based approach. This method is designed to systematically detect controversy in any social digital network discussion, offering a structured and efficient way to analyze online debates.

Their methodology consists of a four-phase pipeline:

  1. Graph Building Phase: In this initial phase, they construct a graph where nodes represent users or pieces of content, and edges represent interactions between them. This graph serves as the foundational structure for further analysis, capturing the relationships and interactions within the discussion.

  2. Community Identification Phase: Once the graph is built, they identify distinct communities within it. This phase helps to reveal clusters of users who share similar viewpoints, which is crucial for understanding the polarization of the discussion.

  3. Embedding Phase: In this phase, they apply embedding techniques to represent the nodes and communities in a continuous vector space. This mathematical transformation allows for the quantification of relationships and similarities between different parts of the network, facilitating more precise analysis.

  4. Controversy Score Computation Phase: Finally, they compute a controversy score based on the embeddings and the identified communities. This score quantifies the level of controversy in the discussion, taking into account factors such as the degree of polarization and the intensity of interactions between opposing groups.

Now let’s dive into each of these stages.

Graph Building


The objective of this phase is to construct a network that accurately represents the activity related to the discussion. This graph serves as the foundational structure for analyzing the dynamics and interactions within the conversation.


For each topic, the researchers build a graph using the following approach:


  • Vertex Assignment: They assign a vertex to each user who contributes to the discussion. Each user is represented as a unique node within the graph.

  • Directed Edges: A directed edge is added from node U to node V whenever user U retweets a tweet posted by user V. This directed edge signifies the flow of information and indicates a relationship between the two users.

They choose retweets because they are typically considered as endorsements of the original tweet. Therefore, the directed edges in the graph reflect not just the interaction but also a level of agreement or support between users.

Community Identification

To accurately identify a community's jargon, it is essential to precisely define its members. The researchers employ a systematic approach to clustering the graph, ensuring that the identified communities reflect the true structure of the discussion.

The researchers utilize the popular Louvain algorithm[4] for clustering the graph. This algorithm is well-regarded for its efficiency and effectiveness in detecting community structures within large networks. As a structure-based approach, it does not require a predefined number of clusters. Instead, it dynamically identifies the optimal community structure based on the network's topology. This adaptability is crucial for capturing the natural divisions within the discussion.

The algorithm's performance is a key factor in its selection for this task. Known for its speed and accuracy, it is particularly well-suited for analyzing large-scale social network discussions, where the volume of interactions can be immense.

After clustering, they focus on the two largest communities. These communities are likely to encompass the most significant and active portions of the discussion, providing a comprehensive view of the primary factions within the debate.


Embedding


In this phase, the researchers aim to embed each user into a corresponding vector representation. This step is critical for transforming the qualitative aspects of the discussion into a quantitative format that can be further analyzed.


They begin by grouping the tweets of users from the two principal communities identified in the previous stage. Each user's tweets are collected and sanitized to remove any irrelevant or potentially harmful content. This ensures that the data used for embedding is clean and representative of the users' true interactions and language patterns.


To estimate the embeddings, the researchers selected two models: FastText and BERT. If you want to know more about them you can check our previous article [5]. They trained both models in a supervised manner, using labeled data to guide the learning process. This approach allows the models to accurately capture the distinctive linguistic features of users within the two principal communities. By embedding users into vector representations, the researchers create a numerical foundation that can be used for subsequent analysis, including the computation of controversy scores. This phase is essential for bridging the gap between raw text data and structured analyzable metrics.


Controversy Score Computation


Here, the researchers aim to quantify the level of controversy within the discussion by analyzing the embeddings of the most influential users. This process involves several key steps to ensure accurate measurement.


First, they identify the top 30% of users with the highest authoritative and hub scores, referring to these as central users. These central users are considered the most influential and pivotal in shaping the discussion. The embeddings of these users are then used for further calculations.


Next, the researchers compute the centroids for the clusters of central users from the two principal communities, C1 and C2. They also compute a global centroid that represents the overall central tendency of all users in the discussion. The centroids serve as reference points for measuring the dispersion of users within each community and across the entire discussion.


To quantify the dispersion, they compute two key metrics: D1 and D2. D1 is the sum of distances from users in cluster C1 to their respective centroid, while D2 is the sum of distances from users in cluster C2 to their centroid. Additionally, they calculate the global distance, D_glob, which is the sum of distances from all users to the global centroid. These distances indicate how tightly or loosely the users in each community are clustered around their central points.


The researchers experiment with several distance metrics to ensure robust measurement. They test Cosine, Euclidean, Mahalanobis, and Manhattan distances to determine which metric best captures the dispersion and polarization within the discussion.


Finally, the r score is computed as:



R score equation

If the dataset forms a single, cohesive cloud of points, the value of r will be close to 1, indicating minimal polarization. However, if the dataset is divided into two distinct clusters, r will be significantly greater than 1, reflecting a high level of controversy. This metric provides a clear numerical assessment of how polarized the discussion is.


In the following picture, we can see examples of the idea of this score.



Centroids and clusters in a controversial and no controversial cases
Centroids and clusters in a controversial and no controversial cases

Results


In the next figure, they present the score distributions of FastText and BERT, using the four different distances described earlier, and compared with previous works (“RW” and “DMC”. They illustrate these distributions using beanplots, with scores from controversial datasets on the left and non-controversial datasets on the right. It is important to note that, by definition, the "DMC" approach yields higher scores for controversial datasets and lower scores for non-controversial datasets, resulting in inverted distributions.


Controversy measures of their metric with different models and distance measures along with other metrics of the state of the art.  If the blue distribution is well separated from the orange one means that the metric can differentiate between controversial and no controversial cases.
Controversy measures of their metric with different models and distance measures along with other metrics of the state of the art. If the blue distribution is well separated from the orange one means that the metric can differentiate between controversial and no controversial cases.

The less overlap there is between the two distributions, the better the pipeline performs. To quantify the performance of the different approaches, they calculate the ROC AUC. By definition, this value ranges between 0 and 1, where 0.5 indicates perfectly overlapping curves (i.e., random scoring), while values of 0 and 1 correspond to perfectly separated distributions. 

As we can see in the picture, the best score (the highest value) is obtained by the FastText model with cosine distance, surpassing previous methods.


Conclusions


The researchers designed an NLP-based pipeline to measure controversy, testing several variants including two embedding techniques (using FastText and BERT language models) and four distance measures. Their best approach, which combines FastText with cosine distance, outperforms the state-of-the-art graph-based methods and their previous work in terms of ROC AUC score and speed.

These results pave the way for a new era of social network analysis, enabling the detection of different viewpoints more quickly and accurately. This advancement has the potential to foster healthier discussions by helping people engage with diverse perspectives in a more constructive manner.


References


[1] de Zarate, J. M. O., Di Giovanni, M., Feuerstein, E. Z., & Brambilla, M. (2020, September). Measuring controversy in social networks through nlp. In International Symposium on String Processing and Information Retrieval (pp. 194-209). Cham: Springer International Publishing.


[2] Garimella, K., Morales, G. D. F., Gionis, A., & Mathioudakis, M. (2018). Quantifying controversy on social media. ACM Transactions on Social Computing, 1(1), 1-27.


[3] Jacomy, M., Venturini, T., Heymann, S., & Bastian, M. (2014). ForceAtlas2, a continuous graph layout algorithm for handy network visualization designed for the Gephi software. PloS one, 9(6), e98679.


[4] Blondel, V. D., Guillaume, J. L., Lambiotte, R., & Lefebvre, E. (2008). Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10), P10008.


Comments


bottom of page