aio-libs/Lobby

Hey guys I need some help using aiohttp

I have a list of proxies to check in the format:

108.30.209.198:80
124.158.167.18:8080
195.154.207.153:80
47.244.235.213:80
192.162.193.243:36910
95.31.5.29:54651
157.230.103.189:36399
86.110.27.165:3128
113.254.178.224:8383
182.160.124.26:8081
203.176.135.102:52234
51.15.196.124:80
190.85.115.78:3128
185.61.176.176:8080
54.159.141.165:80
203.202.245.58:80
51.77.123.244:80

I need to check two things:

What type of proxy it is (http, socks4, socks5)

If it's alive

Currently, the way I'm doing it is with requests, but it's super slow. I want to move to an asynchronous checking method.

What I have now:

for proxy_type in ['http', 'socks4', 'socks5']:
    try:
        proxies = {
          'http': f'{proxy_type}://{proxy}',
          'https': f'{proxy_type}://{proxy}',
        }
        requests.get('https://api.ipify.org', proxies=proxies, timeout=10)
        return proxies, proxy_type
    except:
        pass
return None, None

What I want to do:

Asynchronously do a get request using each proxy, I need there to be a thread count I can control.

Also, asynchronously recheck a proxy using all three different types of proxy protocols.

For example, if our list we're checking is the proxy list above (real list is in the hundreds of thousands) and our thread count is 3 (will most likely be something like 30 in production)

async get request using 108.30.209.198:80 proxy in http mode

async get request using 108.30.209.198:80 proxy in socks4 mode

async get request using 108.30.209.198:80 proxy in socks5 mode

Wait and gather all 3 get request results If all fail remove proxy from list, dead proxy. If one passes add proxy to good proxy list with corresponding proxy protocol

async get request using 124.158.167.18:8080 proxy in http mode

async get request using 124.158.167.18:8080 proxy in socks4 mode

async get request using 124.158.167.18:8080 proxy in socks5 mode

Wait and gather all 3 get request results If all fail remove proxy from list, dead proxy. If one passes add proxy to good proxy list with corresponding proxy protocol

async get request using 195.154.207.153:80 proxy in http mode

async get request using 195.154.207.153:80 proxy in socks4 mode

async get request using 195.154.207.153:80 proxy in socks5 mode

Wait and gather all 3 get request results If all fail remove proxy from list, dead proxy. If one passes add proxy to good proxy list with corresponding proxy protocol

What I have so far, messing around with aiohttp and aiohttp_proxy:

import asyncio
from aiohttp import ClientSession
from aiohttp_proxy import ProxyConnector, ProxyType
async def fetch(url, proxy_type, proxy):
    connector = ProxyConnector.from_url(proxy_type + '://' + proxy)
    try:
        async with ClientSession(connector=connector) as session:
            async with session.get(url, timeout=10) as response:
                return await response.read(), proxy_type
    except:
        return None

async def run(proxy):
    url = "https://api.ipify.org/"
    tasks = []
    for proxy_type in ['http', 'socks4', 'socks5']:
        task = asyncio.ensure_future(fetch(url, proxy_type, proxy))
        tasks.append(task)

    responses = await asyncio.gather(*tasks)
    print(responses)

loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run('183.88.232.207:8080'))
loop.run_until_complete(future)