Make sure all proxy authentication credentials match those on your zone page and that the requests being sent (especially through the API) include all the necessary information. Rather, update your proxy settings to include all the IPs accessing the network within your whitelist. Another cause of a 407 error is found within your proxy settings, such as the necessary IP not having been previously whitelisted or a specific zone you are trying to use being inactive. When using a proxy, this means the credentials you provided are incorrect, your request is missing authorization details, or the crawler being used has not been authenticated with the proxy provider. The site can also respond with a 404 error code, which means ‘Not Found’ and commonly arises when the server doesn’t want to divulge its reason for denying entry.Ī 407 error code refers to a tunnel connection failing or proxy authentication being required. In some cases, the site will provide an explanation, but in others the site may merely respond with a 403 error code itself with no reasoning whatsoever. The request was understood, but the site did not want to grant admittance. An example of this is trying to access a specific profile on a social media site when you are not signed in.Ī 403 error code, however, means your access to the site is forbidden. A 401 error code means you are not authorized to access the target site, and that is why the page will not load. Choosing a specific user-agent provides more detailed information in the request, meaning there is less room for misinterpretation and less chance of the request being redirected.Ī 4XX is a client-side error and is received when the request you sent to the server was misunderstood or inaccurate, resulting in the page being unable to load. If the redirect is occurring due to a lack of information, within the request itself, then this can easily be configured or overcome by specifying a user-agent within your proxy settings. For instance, a 301 error code means a page has been permanently moved and, therefore, you were redirected to the new URL. It means everything is OK and the request has been received by the target site.Ī 3XX error code means you were redirected because your request has multiple responses. HTTP status codes meaningĪ 200 status code is the response you want to attain. Understanding the nature of error codes is the first step to overcoming them. When a request fails, it returns with the respective HTTP error code, which reveals the reason the request was unsuccessful. However, if you are not properly managing your proxies when crawling or scraping, many errors can result in unsuccessful requests. Navigating the online web should be easy. Are your scraping operations coming to a halt and you have no idea why? By simply learning what error codes mean, you can easily automate your IP settings and become a scraping master.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |