0x1949 Team - FAZEMRX - MANAGER
Edit File: robotparser.cpython-310.pyc
o ��+g�$ � @ s\ d Z ddlZddlZddlZdgZe�dd�ZG dd� d�ZG dd� d�Z G d d � d �Z dS ) a% robotparser.py Copyright (C) 2000 Bastian Kleineidam You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2 The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt � N�RobotFileParser� RequestRatezrequests secondsc @ sr e Zd ZdZd dd�Zdd� Zdd� Zd d � Zd d � Zd d� Z dd� Z dd� Z dd� Z dd� Z dd� Zdd� ZdS ) r zs This class provides a set of methods to read, parse and answer questions about a single robots.txt file. � c C s2 g | _ g | _d | _d| _d| _| �|� d| _d S )NFr )�entries�sitemaps� default_entry� disallow_all� allow_all�set_url� last_checked��self�url� r �)/usr/lib/python3.10/urllib/robotparser.py�__init__ s zRobotFileParser.__init__c C s | j S )z�Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. )r �r r r r �mtime% s zRobotFileParser.mtimec C s ddl }|� � | _dS )zYSets the time the robots.txt file was last fetched to the current time. r N)�timer )r r r r r �modified. s zRobotFileParser.modifiedc C s& || _ tj�|�dd� \| _| _dS )z,Sets the URL referring to a robots.txt file.� � N)r �urllib�parse�urlparse�host�pathr r r r r 6 s zRobotFileParser.set_urlc C s� z t j�| j�}W n@ t jjyI } z2|jdv r d| _n|jdkr0|jdk r>d| _W Y d}~dS W Y d}~dS W Y d}~dS W Y d}~dS d}~ww |� � }| � |� d�� � � dS )z4Reads the robots.txt URL and feeds it to the parser.)i� i� Ti� i� Nzutf-8) r �request�urlopenr �error� HTTPError�coder r �readr �decode� splitlines)r �f�err�rawr r r r" ; s ����zRobotFileParser.readc C s2 d|j v r| jd u r|| _d S d S | j�|� d S �N�*)� useragentsr r �append)r �entryr r r � _add_entryH s �zRobotFileParser._add_entryc C sJ d}t � }| �� |D �] }|s(|dkrt � }d}n|dkr(| �|� t � }d}|�d�}|dkr7|d|� }|�� }|s>q |�dd�}t|�dk�r|d �� �� |d< tj � |d �� �|d< |d dkr~|dkrs| �|� t � }|j � |d � d}q |d dkr�|dkr�|j � t|d d �� d}q |d d kr�|dkr�|j � t|d d �� d}q |d d kr�|dkr�|d �� �� r�t|d �|_d}q |d d k�r|dk�r|d �d�}t|�dk�r|d �� �� �r|d �� �� �rtt|d �t|d ��|_d}q |d dk�r| j� |d � q |dk�r#| �|� dS dS )z�Parse the input lines from a robots.txt file. We allow that a user-agent: line is not preceded by one or more blank lines. r r � �#N�:z user-agent�disallowF�allowTz crawl-delayz request-rate�/�sitemap)�Entryr r- �find�strip�split�len�lowerr r �unquoter* r+ � rulelines�RuleLine�isdigit�int�delayr �req_rater )r �lines�stater, �line�i�numbersr r r r Q sv � � � � �� �zRobotFileParser.parsec C s� | j rdS | jr dS | jsdS tj�tj�|��}tj�dd|j|j |j |j f�}tj� |�}|s3d}| j D ]}|�|�rD|�|� S q6| jrN| j�|�S dS )z=using the parsed robots.txt decide if useragent can fetch urlFTr r3 )r r r r r r r; � urlunparser �params�query�fragment�quoter � applies_to� allowancer )r � useragentr � parsed_urlr, r r r � can_fetch� s( � � zRobotFileParser.can_fetchc C �> | � � sd S | jD ] }|�|�r|j S q | jr | jjS d S �N)r r rL r@ r �r rN r, r r r � crawl_delay� � �zRobotFileParser.crawl_delayc C rQ rR )r r rL rA r rS r r r � request_rate� rU z RobotFileParser.request_ratec C s | j sd S | j S rR )r r r r r � site_maps� s zRobotFileParser.site_mapsc C s, | j }| jd ur|| jg }d�tt|��S )Nz )r r �join�map�str)r r r r r �__str__� s zRobotFileParser.__str__N)r )�__name__� __module__� __qualname__�__doc__r r r r r" r- r rP rT rV rW r[ r r r r r s I c @ s( e Zd ZdZdd� Zdd� Zdd� ZdS ) r= zoA rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.c C s< |dkr|sd}t j�t j�|��}t j�|�| _|| _d S )Nr T)r r rG r rK r rM )r r rM r r r r � s zRuleLine.__init__c C s | j dkp |�| j �S r( )r � startswith)r �filenamer r r rL � s zRuleLine.applies_toc C s | j rdndd | j S )N�Allow�Disallowz: )rM r r r r r r[ � s zRuleLine.__str__N)r\ r] r^ r_ r rL r[ r r r r r= � s r= c @ s0 e Zd ZdZdd� Zdd� Zdd� Zdd � Zd S ) r5 z?An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _d | _d | _d S rR )r* r< r@ rA r r r r r � s zEntry.__init__c C s� g }| j D ] }|�d|� �� q| jd ur |�d| j� �� | jd ur3| j}|�d|j� d|j� �� |�tt| j �� d� |�S )Nz User-agent: z Crawl-delay: zRequest-rate: r3 � ) r* r+ r@ rA �requests�seconds�extendrY rZ r< rX )r �ret�agent�rater r r r[ � s z Entry.__str__c C sF |� d�d �� }| jD ]}|dkr dS |�� }||v r dS q dS )z2check if this entry applies to the specified agentr3 r r) TF)r8 r: r* )r rN ri r r r rL � s �zEntry.applies_toc C s$ | j D ] }|�|�r|j S qdS )zZPreconditions: - our agent applies to this entry - filename is URL decodedT)r< rL rM )r ra rD r r r rM s �zEntry.allowanceN)r\ r] r^ r_ r r[ rL rM r r r r r5 � s r5 ) r_ � collections� urllib.parser �urllib.request�__all__� namedtupler r r= r5 r r r r �<module> s B