365 Data Science

Usually in any K-means clustering problem, the first problem that we face is to decide the number of clusters(or classes) based on the data. This problem can be resolved by 3 different metrics(or methods) that we use to decide the optimal ‘k’ cluster values. They are:

1. Elbow Curve Method
2. Silhouette Score
3. Davies Bouldin Index

Let us take a sample dataset and implement the above mentioned methods to understand their working.

We will use the make blobs dataset from sklearn.datasets library for illustrating the above methods

Now let’s look at what these methods area and that after implementing those three methods on the created dataset what are the results.

A) Elbow Curve
The main idea is to define clusters such that the total within-cluster sum of square (WSS) is minimized. It measures the compactness of the clustering and we want it to be as small as possible. The idea is to choose a number of clusters (k) so that adding another cluster doesn’t improve much better the total WSS.

***What Within-cluster Sum of Square mean (WSS)?
Basically it is the sum of squared distance (usually Euclidean distance) from it’s nearest centroid (center point of cluster).

It decreases with increasing number of clusters(k). Aim is to find the bend (like an elbow joint) point in the graph.

Trending AI Articles:

1. Fundamentals of AI, ML and Deep Learning for Product Managers

2. The Unfortunate Power of Deep Learning

3. Graph Neural Network for 3D Object Detection in a Point Cloud

4. Know the biggest Notable difference between AI vs. Machine Learning

***This method is called elbow curve because the visual representation of the WSS w.r.t. the number of clusters(k) looks like human elbow

Here, we find that the k=3 is the bend(elbow) point.
*** Usually elbow curve method is a little ambiguous as the bend point for some datasets is not visible clearly

B) Silhouette Score
Silhouette Score is calculated using mean of intra-cluster distance (a) and the mean of nearest-cluster distance (b) for each sample. The Silhouette Coefficient for a sample is (b -a) / max(a, b). For better clarification, intra-cluster distance (a) is distance of sample point to it’s centroid and (b) is distance of sample point to nearest cluster that it is not a part of. Hence, we want the silhouette score to be maximum. Thus, have to find a global maxima for this method.

Silhouette coefficient exhibits a peak characteristic as compared to the gentle bend in the elbow method. This is easier to visualize and reason with.

Here, we can easily visualize the peak point at k=3.

C) Davies Bouldin Index
It is defined as a ratio between the cluster scatter and the cluster’s separation. Basically a ratio of within-cluster distance and between cluster distances. Aim is to find optimal value in which clusters are less dispersed internally and are farther apart fro each other (i.e. distance between two clusters is high). Hence, a lower value of Davies Bouldin index will mean that the clustering is better.

As I mentioned earlier lower value is desired, so we find the global minima point i.e. k= 3.

So after using all the above mentioned methods, we concluded that optimal value of ‘k’ is 3. Now, implementing the k-means clustering algorithm on the dataset we get the following clusters.

We can see from the above graph that all points are classified into three clusters appropriately. Hence, the k=3 was an optimal value for clustering.

You can find the entire code snippet at GitHub location here: https://github.com/kavyagajjar/K-Means-Clustering/blob/main/K-means-Make_blobs/Optimal_K_using_k_means_clustering.ipynb

Don’t forget to give us your ? !

3 minute read to ‘How to find optimal number of clusters using K-means Algorithm’ was originally published in Becoming Human: Artificial Intelligence Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.

Via https://becominghuman.ai/3-minute-read-to-how-to-find-optimal-number-of-clusters-using-k-means-algorithm-eaa6bdce92cc?source=rss—-5e5bef33608a—4

A new experiment from the Evolve team

Human + AI is a powerful combo

In the future, I see operators guiding groups of algorithms: it will be a time when machines will be able to generate their own knowledge based on human-produced frameworks … Human plus machine means finding a better way to combine better interfaces and better processes

— Gary Kasparov (Chess Grandmaster)

There is a lot of value in using intelligent systems to improve the quality of our work. For text analysis our best AI algorithms are still lacklustre (in spite of what some headlines may tell you about work like GPT-3). Perhaps one day an algorithm could take all of your qualitative research notes and summarise them for you. For now we have to rely on other approaches.

At Evolve we’ve been looking at using Machine Learning to speed up the process of analysing UX research. Our latest approach is to use Natural Language Processing to kickstart the research process rather than to try to replace it.

We’ve experimented with a lot of approaches including complex algorithms involving neural networks. The complex approaches didn’t generate particularly impressive results and we’ve found that a simple algorithm actually worked best.

Our idea

Our approach is simple:

1. Use Natural Language Processing techniques to identify the most important keywords
2. Create a group of keywords that are related
3. Show all of the research notes that contain one of those keywords
4. Allow researchers to modify the keyword groups or create their own
5. The researcher can select groups of notes that contain that keyword and add them to a research finding

The algorithm isn’t perfect. But given that the algorithm can analyse a typical UX research project in a couple of seconds its results are generally useful.

The most useful aspect of this approach is that it can help fast-track the analysis process. We want users to be actively engaged in the research process rather than relying on an algorithm to analyse their research for them.

If the algorithm adds incorrect keywords to a group — remove them. If it misses an important keyword — add it. If there is a topic that you feel isn’t covered — create one.

Trending AI Articles:

1. Fundamentals of AI, ML and Deep Learning for Product Managers

2. The Unfortunate Power of Deep Learning

3. Graph Neural Network for 3D Object Detection in a Point Cloud

4. Know the biggest Notable difference between AI vs. Machine Learning

From there it’s about finding patterns and interesting observations in the data. With this approach it should be easier to identify research findings by only looking at a subset of the data.

Is this approach effective?

At this stage we’re looking for feedback on whether or not this is useful in analysing the UX research process. The Evolve UX Research app has a completely free plan and our paid plan is free while we’re in beta. If you have any feedback on this approach we’d love to hear it.

Don’t forget to give us your ? !

Can machine learning speed up UX research analysis? was originally published in Becoming Human: Artificial Intelligence Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.

Via https://becominghuman.ai/can-machine-learning-speed-up-ux-research-analysis-22ffd38a4c2d?source=rss—-5e5bef33608a—4

“Perl was designed to work more like a natural language. It’s a little more complicated but there are more shortcuts, and once you learned the language, it’s more expressive.”

When I initially heard of the term Natural Language Processing, I was like “Wait what….. how is it possible that machines can understand and interpret what we say”. Then after a few google searches, I came to know that it is nothing but a lot of text data and maths.

Natural Language Processing is divided into:-

• Morphological and Lexical Analysis.
• Syntactic Analysis.
• Semantic Analysis.
• Sentiment Analysis.
• Pragmatic Analysis.

So, Today we are going to talk about the Easiest one “Sentiment Analysis”.

Sentiment Analysis is an analysis technique to get the emotions of the person by the text he wrote.

Eg:-

“This Place is not Good” This line has a Negative Sentiment.

“This Place is Good” has a Positive Sentiment.

So Predicting these Sentiments is called Sentiment Analysis. It can be of more types like happy or not happy, Happy Angry or Surpised.

Coming to The Practical Part:-

1. There are various steps in Data Preprocessing for Natural Language Processing:-

→ First is the Collection of data.

→ Removing the Punctuations in the Text, But removing punctuations is not always good because it also carries emotions in text, so it totally depends upon the user and data what is it representing.

→ Removing Stop Words from the Text like “and, or, etc”, as it doesn’t carry any sentiment with it.

→ Stemming of the data, Stemming of text is “Stemmers remove morphological affixes from words, leaving only the word stem.”

eg:- Working, Worked, Work →(Stemming) →Work

here you can see that all the words carry the same meaning with it so including all the terms in our dictionary will make our model slow, instead, we can stem the words and replace them with their stemmed words.

Trending AI Articles:

1. Fundamentals of AI, ML and Deep Learning for Product Managers

2. The Unfortunate Power of Deep Learning

3. Graph Neural Network for 3D Object Detection in a Point Cloud

4. Know the biggest Notable difference between AI vs. Machine Learning

Here Your Data PreProcessing is Complete…….

From here 60% of your work is complete.

Now What you have to do Is to Create a Frequency Table of Repetition of words in your text. Don’t worry Tfidf Vectorizer will do the work for you in one line of code just initialize it and run it with your text data.

Here this will create your Frequency table of words with the Sentiment it carries, Create a Pipeline with your preferred Classification Technique, and Tfidf Vectorizer.

Train the Model with your Text Data and Check With Test Data….. That’s all. 🙂

My Jupyter Notebook for This Sentiment Analysis:-

harsh674/Sentiment-Analysis-of-Financial-News

Don’t forget to give us your ? !

Sentiment Analysis in 3 Minutes, Intro to Natural Language Processing. was originally published in Becoming Human: Artificial Intelligence Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.

Via https://becominghuman.ai/sentiment-analysis-in-3-minutes-intro-to-natural-language-processing-cd08ae3206eb?source=rss—-5e5bef33608a—4