Behind the talk of making AI ethical, lies the task of making ethical AI by directly working with & listening to the communities it…
Continue reading on Becoming Human: Artificial Intelligence Magazine »
Neural Networks From Scratch Using Python
what is Neural Network?
Neural Networks are inspired by biological neuron of Brain
from the dendrites inputs are being transferred to cell body , then the cell body will process it then passes that using axon , this is what Biological Neuron Is .
Same process like Brain Neuron
- Inputs are passed
- + Symbol in the cell body denotes adding them together
- Threshold is Activation Function (We will talk that later)
How Neural Network Works ?
Steps :
- Takes the input Values
- Multiplies with the weight adding bias value to it
- Forward Propagation is finished
- now check the error
- then change the Weight values
- Back-propagation is Finished
- Repeat it until error gets low as possible
What is Activation Function ?
IF we did not use the activation function means it is equal to the Linear Regression Model ,
Non-Linear activation Function are more overly used because in real world data-set we will handle non linear data-sets a lot so that linear is not much usefull
Activation function are used in the hidden layer and output layer
there are many Non-linear Activation functions are available like Sigmoid , tanh , ReLU etc….
want to know more about activation function
Each Activation functions are having their own derivatives
In this Sigmoid Derivatives has been shown , Derivatives are used for updating the Weights
Trending AI Articles:
1. Only Numpy: Implementing Convolutional Neural Network using Numpy
3. Understanding and building Generative Adversarial Networks(GANs)- Deep Learning with PyTorch
if your problem Regression means in the hidden layer and output layer you should not use Sigmoid you can use ReLU , Classification means Sigmoid we can use
We are having the inputs in which 0 and 1 , outputs are 0 and 1 , so for these we can use sigmoid
if you have new situation like for 1 0 0 what will be the output ?
Forward Propagation
Takes the input and multiplying weights with it , adding bias to it , pass it into sigmoid function y is calculated , then doing subtraction with original y to calculated y
error =original_y — calcul_y
calculated y is passed to the sigmoid derivatives stored as sd , then multiplying error and sd , and then doing the matrix multiplication and storing the value as Adju .
Backward Propagation
Then adding Adju values to the weight , the weight has been updated , weight+=adju
repeat it until the error gets low
Code : git-hub
- In_init_ function setting up the weight randomly
- Defining sigmoid and its derivatives
- the Backpropagation steps in train function
- think function just passes the values to the neural network
Randomly started weights and finally error corrected weights , then we have given iput as 1 0 0
output for it 0.9993704 nearly 1 almost right , this is how the neural networks are Working
Don’t forget to give us your ? !
Neural Networks From Scratch Using Python was originally published in Becoming Human: Artificial Intelligence Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.
Build PyTorch Models Easily Using torchlayers
torchlayers aims to do what Keras did for TensorFlow, providing a higher-level model-building API and some handy defaults and add-ons useful for crafting PyTorch neural networks.
Originally from KDnuggets https://ift.tt/2yHZl17
What Are Request Headers And How to Deal with Them When Scraping
Hello, fellow scrapers! In this tutorial, we will talk about request headers – what they are, in what way they may restrict our scraping and how to use them to our advantage. We will also discuss a special type of a request header – cookies.
So, what is a header in the context of a request?
Well, when you’re sending a request to a server you’re not just saying: ‘Hey, give me that info, please’. You are also providing information about the request itself – information, such as the encoding and language of the expected response, the length and type of data provided, who is making the request and so on. These pieces of information, referred to as headers, are intended to make communications on the web easier and more reliable, as the server has a better idea of how to respond.
Okay. But the question remains – how are these headers specified?
Well, every type of header information is contained in a standardized header field. Two of the most common header fields are the ‘User-Agent’ and ‘cookie’. Let’s take a deeper look into those.
Request Headers: What is a user agent string?
When a software sends a request, it often identifies itself, its application type, operating system, software vendor, or software version, by submitting a characteristic identification string. This string is referred to as a “user agent string”. You can think of it as an ID card containing some basic information.
All browsers, as well as some popular crawlers and bots, such as ‘google bot’, have a unique ‘user agent string’ that they identify themselves with.
So how does this concern us, the scrapers?
Well, a lot of companies set up their servers in a way that allows them to identify the browser a client is using. In fact, most websites may look a tiny bit different in Chrome, Firefox, Safari and so on. Based on the browser, a specific version of the web page is sent to the client for optimal visual and computational performance. However, this may become an issue for us if we do not provide a proper ‘user agent string’.
There are two things that can happen in that case.
First of all, the server may be set up to send a default variant of a page if it doesn’t recognize the user agent.
In this situation, the HTML we are looking at in our browser may be different from what we receive as a response. The solution in this case, though, is pretty straightforward – exporting the HTML response to a local file and inspecting that one, instead of the browser version.
A more serious issue arises when the server decides to block all unrecognized traffic.
In that case, to continue scraping, we need to provide a legitimate user agent. Fortunately, all browsers’ user agent strings are available publicly on the internet. Thus, we can easily pretend to be a browser.
Let’s see how to do this in Python using the ‘requests’ package.
Incorporating different headers using ‘requests’ is actually a very simple job. All we have to do is supply them in a dictionary format to the ‘headers’ parameter.
For instance, suppose we want to make a GET request to YouTube, pretending to be a client using Chrome. First, we need to find the User-Agent string of Chrome. A quick Google search yielded us this string:
“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36”
Okay. We can save it in a dictionary with field name ‘User-Agent’.
Now, we can make a GET request using the usual ‘get()’ method of the package. The URL of the site we want to connect to is passed as a parameter. Normally, that would be all.
However, in order to incorporate the request headers, we can add them in dictionary form to the additional ‘headers’ parameter. In our case, we have saved the dictionary in the ‘header’ variable, so we pass that to the parameter.
That’s it. This request now contains a ‘User-Agent’ header field.
And here’s the full code:
By adding different fields to our dictionary, we can incorporate different headers into the request.
Dealing with Request Headers: What about the cookies?
If dealing with request headers is that simple, what is so special about the cookies?
Well, an HTTP cookie is a special type of request header that represents a small piece of data sent from a website and stored on the user’s computer. It is different from other headers, as we are not the ones to choose it – it is the website that tells us how to set this field. Then, the cookie can be sent along with subsequent client requests.
Cookies were designed to be a reliable mechanism for websites to remember stateful information, such as items added in the shopping cart in an online store, or to record the user’s browsing activity.
They can also be used to remember arbitrary pieces of information that the user previously entered into form fields, such as names, addresses, passwords, and credit-card numbers.
Cookies perform essential functions in the modern web.
Perhaps the most important one is the authentication cookie. It is the most common method used by web servers to know whether the user is logged in or not, and which account they are logged in with. This basically means that you are not required to sign in every time you open or reload a page.
Cookies are implemented a bit different from the ‘user agent’, as websites usually tell us how to set them the first time we visit a page.
Despite that, the awesome ‘requests’ package saves the day once again with its session class.
We can open a new session using this command.
Notice that we assign the session to a variable. Later, we can make requests through this variable. Every such request within that session will incorporate persistent cookies automatically. We don’t have to do anything. After we are done, we have to close the session.
Here is an example code:
Just remember that the request should be made through the session variable. You can find more about that in the official ‘requests’ documentation here.
You have now added the request headers weapon to your web scraping arsenal. Let us know how you implemented it in your practice in the comments below!
Eager to become a Web Scraping Pro? Check out the 365 Web Scraping and API Fundamentals in Python Course!
The course is part of the 365 Data Science Program. You can explore the curriculum or sign up 12 hours of beginner to advanced video content for free by clicking on the button below.
The post What Are Request Headers And How to Deal with Them When Scraping appeared first on 365 Data Science.
from 365 Data Science https://ift.tt/3aZtIhO
24 Best (and Free) Books To Understand Machine Learning; COVID-19 Visualized: The power of effective visualizations; 20 AI DS ML terms you need to know
Also: 20 AI, Data Science, Machine Learning Terms You Need to Know in 2020 (Part 2); Linear to Logistic Regression, Explained Step by Step.
Originally from KDnuggets https://ift.tt/2UWv3QQ
10 Must-read Machine Learning Articles (March 2020)
This list will feature some of the recent work and discoveries happening in machine learning, as well as guides and resources for both beginner and intermediate data scientists.
Originally from KDnuggets https://ift.tt/2XnG13y
Top KDnuggets tweets Apr 01-07: How to change global policy on #coronavirus
Also: 10 Must-read Machine Learning Articles (March 2020); Mathematics for Machine Learning: The Free eBook; Free Mathematics Courses for Data Science & Machine Learning; 9 Best YouTube Playlists and Videos — #Python for #MachineLearning
Originally from KDnuggets https://ift.tt/3e0yOMV
How to Do Hyperparameter Tuning on Any Python Script in 3 Easy Steps
With your machine learning model in Python just working, it’s time to optimize it for performance. Follow this guide to setup automated tuning using any optimization library in three steps.
Originally from KDnuggets https://ift.tt/3aVzFfL
Computerised Robots and Sensory Experiences
The American philosopher John Searle often makes the point that many believers in Artificial Intelligence (AI), computational cognitive…
Continue reading on Becoming Human: Artificial Intelligence Magazine »
The Art of Tackling Large Problems
There is no single day in the life of a software engineer when it comes to tackling complex and large problems in a fast-paced tech environment. Before delving into details of solving problems of complex nature, let’s have a look at the process of problem-solving in general.
At a high level, there are six phases to tackle a large scale problem:
- Identify the opportunity
- Analyze and look for patterns
- Define a high-level strategy
- Create simple, fast solutions
- Deliver, quantify, and communicate
- Refine the vision and scale the solution
We will go over the each of the six phases in detail and talk about:
- The objective of each phase
- Duration and typical actions taken
- Goals we set and expected outcome before move on to the next phase
Rendering reliability will be used as a working example to give more color and depth to each phase for software engineers.
Software engineers are not short of opportunities for problem-solving at the companies. The first step is to eliminate the noise and identify the fundamental problem(s) that is most impactful to the business. In this step, you validate the list of opportunities with additional supporting data points, that include SEVs, SLA tasks, logging data.
There is no set time duration for this phase; looking out for large class of issues (rather than fixing constant small bugs) should be part of our day-to-day operation.
Let’s take the following example of identifying rendering reliability opportunity
Product teams have usually been relying on manual QA tests to prevent incorrect rendering online ads. However, a large number of rendering variations based on device, platform, and ad product types can make it hard to ensure reliability. Even small corner cases can have a significant revenue impact, and these cases are usually introduced by ad or non-ad teams since rendering is built on top of a large shared codebase. Since these could impact advertiser trust, revenue loss and engineering productivity, rendering reliability stands out to be a top problem for a tech company.
Expected outcome before moving to next phase will be to find one or few investments that can give high impact for the business. Moving to next phase without addressing this may yield into many parallel analysis, which will be very time consuming. Based on the example above, rendering reliability is one of the key investment that could have high business impact (improve advertiser trust, reduction of refund, avoid bad PR that may tarnish company’s reputation).
It is often tempting to avoid time for further investigation as everyone wants to deliver a solution and show impact as soon as possible. Before diving into a solution, there should be an identification of a few critical subsets of problems that make up the bigger problem. Not investing time on this phase may lead to solutions that aren’t highly impactful.
Time boxing this phase would help avoid delving too much into the tail-end issues. This phase usually takes 1–2 months. Typically we set “understand” goals and aim for developing conviction for the problem space and the impact we can generate by eliminating it.
No one would prefer to jump-in and start fixing problems by picking one Product or one Surface or fixing problems for one format at a time. Rather, time should be spent in analyzing classes of problems, impacts, recurrence using data.
Given the case of rendering reliability, below are some examples of major issues which are commonly encountered in tech companies:
- Cropping and sizing errors: These include cases where images were incorrectly cropped, resized or zoomed, or thumbnails for video ads not correctly rendered.
- Missing components or empty attachment: These include cases where attachment is empty, attachment has missing components (partial rendering) or images don’t load.
- Wrong content: These include cases where wrong creatives or thumbnails from other ad campaigns or organic posts are rendered for an ad.
Expected outcome before moving to next phase should be to dissect the opportunity further into classes of problems. From the above example, it was clear that focusing on image cropping, wrong image and missing component will yield higher success.
Instead of doing a tactical solution of fixing each problem on isolation, you can invest a few weeks in delving into strategies in solving the classes of problems. This includes analyzing possible ways to solve the problem, investigating the tradeoffs of different approaches, brainstorming with broader team and coming with a unified approach. It is equally important to identify key metrics that will measure the impact, although it is difficult to bet on goals during this phase. At this phase we slow down a bit in order to run faster.
Let’s take a deep dive into rendering pipeline, understand the complexities, touch points, code complexity, and potential stages the problems can be detected and prevented. We can broadly group the opportunities in three major stages as below:
- Development stage detection — Detect rendering issues at the time of coding/development
- Pre-rendering detection & prevention — Detect & prevent rendering issues at the time of ads delivery
- Post-rendering detection — Detect rendering issues on the client-side, validate the final rendering
Expected outcome before moving to next phase should be having a clear strategy on how to develop a solution is an important outcome before we come-up with a solution. In the above example, we can create a three phase strategy to improve rendering reliability. We then can create three smaller teams to start working towards a solution.
Once the team start producing impact, the team can incrementally add additional layers of reliability, scalability, and monitoring.
Having concrete goals that define the failure and success states is critical. The solution will most likely won’t cover the whole problem domain yet, however, parts of the problem domain covered should yield results that will help you build confidence in your solution.
Trending AI Articles:
The main goal is to create a solution that can help us succeed fast of fail fast. Although, scalability is one of the considerations, it isn’t the primary goal for many tech companies:
- For the detection during the development phase, we can start with screenshot testing with automated email alert when mismatch occurs and have the engineer manually analyze the emails, tasks and evaluate the business value.
- For pre-rendering detection, we can leverage logging, manually analyze the false-positives, create rules to detect rendering problems at the server-side. This process can be laborious. After adding some rules, we can come up with a solution that helped us shorten the dev cycle.
- For post-rendering detection, we can enable client-side logging, analyze those logs to identify rendering problems, eliminate false-positives, and add rules.
Expected outcome before moving to next phase should be to demonstrate simple, working solution that help address the problem. The solution at this phase is neither highly mature, nor scalable. However, it is important to have basic building blocks to make it a working solution. This includes monitoring and test automation. In the above example, we can create three simple solutions just to prove that we are able to detect and prevent rendering problems.
Creating the right communication channels and cadence with customers, leadership and partners will help increase the visibility of the work and allow critical feedback to flow in. It is vital to have key metrics to measure the impact and make them a prominent part of the communication. The following are some examples of such improvement points:
- Rolling out some new rules to detect rendering problems
- Improved rendering problem detection impact
- Increasing partnership with related parties to increase traffic coverage for the existing rules
Expected outcome before moving to next phase should be a clear set of metrics to measure the impact, regular consistent communication with partners and leadership. In the rendering reliability example above, there can be some types of bi-weekly and monthly communication channels among partners and leaders.
As the solution gets to a mature state, ongoing investment can be reduced, however, adequate monitoring should be in place to look out for potential regressions. In many tech companies, systems usually evolve quickly, which may require adjusting or re-thinking the solution.
In the rendering reliability example, we can figure out the end goal, prioritize them to get to the completion and reach out to other partners to leverage the solution.
The sample chart below displays how the number of discovered issues can increase at the early stages while adding more detection. Once a solution that can help prevent issues is in place, the number starts decreasing. Increasing the coverage of the solution will usually push the number further down to a state where the problem is not “big” anymore.
Thank you for reading so far! One take-away from this article should be that given these tips and tricks, engineers should really no longer worry about tackling complex problems. So, let’s nail down your next challenge!
Don’t forget to give us your ? !
The Art of Tackling Large Problems was originally published in Becoming Human: Artificial Intelligence Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.
Via https://becominghuman.ai/the-art-of-tackling-large-problems-fe559bc90c3e?source=rss—-5e5bef33608a—4
source https://365datascience.weebly.com/the-best-data-science-blog-2020/the-art-of-tackling-large-problems
365 Data Science 
