elazarl/goproxy

An HTTP proxy library for Go

Almog Baku

@AlmogBaku

cc: @elazarl

Elazar Leibovich

@elazarl

Thanks @AlmogBaku , Thanks. I'm glad the longest request is only 50% longer than squid.

I'd consider using https://github.com/giltene/wrk2 for a better latency analysis by Gil Tenne

I'll try to mention it in the wiki

Almog Baku

@AlmogBaku

@elazarl on the previous test it was faster... BUT unfortunately it seems like my test was wrong

the goproxy had lots of missed requests.. im seeing from the console this:

2016/03/09 12:31:51 [086] WARN: Error copying to client: read tcp 172.31.25.102:8080->198.27.114.146:34600: read tcp 172.31.25.102:49891->204.79.197.200:80: read: connection reset by peer
2016/03/09 12:31:52 [085] WARN: Error copying to client: read tcp 172.31.25.102:8080->198.27.114.146:34594: read tcp 172.31.25.102:49890->204.79.197.200:80: read: connection reset by peer

and

2016/03/09 12:32:29 [118] WARN: Error copying to client: read tcp 172.31.25.102:8080->158.69.118.48:50948: write tcp 172.31.25.102:8080->158.69.118.48:50948: use of closed network connection
2016/03/09 12:32:29 [117] WARN: Error copying to client: read tcp 172.31.25.102:8080->158.69.118.48:50646: write tcp 172.31.25.102:8080->158.69.118.48:50646: use of closed network connection

and

2016/03/09 13:12:01 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms
2016/03/09 13:12:01 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms
2016/03/09 13:12:01 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 10ms
2016/03/09 13:12:02 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms
2016/03/09 13:12:02 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms

I executed the goproxy from root, and exectue ulimit -n 10240 before :|

open files while the stress test is around 1024:

root@ip-172-31-25-102:/home/ubuntu# lsof -p 3946|wc -l
1024

open files after is 13:
root@ip-172-31-25-102:/home/ubuntu# lsof -p 3946|wc -l 13

limit:

root@ip-172-31-25-102:/home/ubuntu# ulimit -n
10240

Almog Baku

@AlmogBaku

ahhhh... i need to run the ulimt from the same tty of the goproxy!

now it worked.. and the results are still better than squid.. but i still had few errors of the tcp write...

untu@ip-172-31-25-103:~$ ab -c 700 -n 40000 -X go.proxy:8080 http://tapuz.co.il/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking tapuz.co.il [through go.proxy:8080] (be patient)
Completed 4000 requests
Completed 8000 requests
Completed 12000 requests
Completed 16000 requests
Completed 20000 requests
Completed 24000 requests
Completed 28000 requests
Completed 32000 requests
Completed 36000 requests
Completed 40000 requests
Finished 40000 requests


Server Software:        
Server Hostname:        tapuz.co.il
Server Port:            80

Document Path:          /
Document Length:        233 bytes

Concurrency Level:      700
Time taken for tests:   19.765 seconds
Complete requests:      40000
Failed requests:        0
Non-2xx responses:      40000
Total transferred:      20600000 bytes
HTML transferred:       9320000 bytes
Requests per second:    2023.76 [#/sec] (mean)
Time per request:       345.892 [ms] (mean)
Time per request:       0.494 [ms] (mean, across all concurrent requests)
Transfer rate:          1017.81 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   16  15.1     12      70
Processing:   107  325 124.5    299    1100
Waiting:      107  325 124.5    299    1100
Total:        107  341 128.0    315    1117

Percentage of the requests served within a certain time (ms)
  50%    315
  66%    386
  75%    429
  80%    457
  90%    520
  95%    574
  98%    636
  99%    681
 100%   1117 (longest request)

only those three errors

root@ip-172-31-25-102:/home/ubuntu# ./goproxy-basic 
2016/03/09 13:28:29 [180] WARN: Error copying to client: read tcp 172.31.25.102:8080->188.68.224.51:32895: write tcp 172.31.25.102:8080->188.68.224.51:32895: use of closed network connection
2016/03/09 13:29:34 [073] WARN: Error copying to client: read tcp 172.31.25.102:48632->212.74.50.1:443: write tcp 172.31.25.102:48632->212.74.50.1:443: use of closed network connection
2016/03/09 13:31:08 [081] WARN: Error copying to client: read tcp 172.31.25.102:8080->188.68.224.51:43394: write tcp 172.31.25.102:8080->188.68.224.51:43394: use of closed network connection

Almog Baku

@AlmogBaku

@elazarl wrk2 doesn't support proxy out of the box...

Almog Baku

@AlmogBaku

@elazarl do you have any idea regarding: elazarl/goproxy#149 ?

Almog Baku

@AlmogBaku

see golang/go#15015

Alex Scotton

@Illizian

Hi, please excuse my ignorance. This is my first foray into Golang. I am trying to write a custom ReqCondition func. I unfortunately lack the knowledge to understand why it doesn't work.

Here is the prototype func:

func isBlacklisted() goproxy.ReqCondition {
  return goproxy.ReqConditionFunc(func(req *http.Request, ctx *goproxy.ProxyCtx) bool {
    return true;
  })
}

and I am including thusly:

proxy.OnRequest(isBlacklisted()).HandleConnect(goproxy.AlwaysReject)

the isBlacklisted func is being called but the ReqConditionFunc is not, I have also tried that without the wrapper

Alex Scotton

@Illizian

scrap that! the issue this whole time was my use of goproxy.AlwaysReject , Pretty sure I'm doing it wrong :P as I'm already useing HandleConnect with an AlwaysMitM function, can you only have one HandleConnect statement?

ZiYuan

@Bpazy

Hi, I have a problem:

func handler(r *http.Request, ctx *goproxy.ProxyCtx) (*http.Request, *http.Response) {
    temp := make([]byte, 1024)
    r.Body.Read(temp) // How to seek to start?
    return r, nil
}

Sriharsha Setty

@horsey

Hello!

Sriharsha Setty

@horsey

@elazarl I came across goproxy while looking around for a CORS proxy server. Basically I am looking at a way to proxy requests for servers that don't handle the CORS very well.

Sriharsha Setty

@horsey

Can I setup goproxy to accept requests in a manner: www.goproxy.com/http://api-server.com to proxy the request to http://api-server.com?

GNU

@cuu

hello all, I just wonder if anyone has asked, how to get the raw request strings from this goproxy ? espcially in HandleConnectFunc with CONNECT MOTHOD...

Odohi David

@spankie

:point_up: December 7, 2016 8:49 PM this proxy will help you with your CORS issue.

ilk

@ilk

Hi - How can access the https response body with goproxy? I have some jobs which fetch xml and json (written in different langs) and they should use a central proxy which save all the responses. I need that for logging/history purposes.

Mohamed Isoukrane

@misoukrane

Hi, I am trying to proxy HTTPS through connect without using MITM, all what I see in the transparent example is AlwaysMitm. is there a different option that I can use to achieve proxying HTTPS without mitm? Thanks in advance.

tsjnsn

@tsjnsn

Do I need to do anything to enable keep alive?

tsjnsn

@tsjnsn

Guess so

func responseIsKeepAlive(res *http.Response) bool {
      // TODO: implement.  for now just always shutting down the connection.

SuperFashi

@hanbang-wang

Hi, one question. Why some of the requests are invisible in OnRequest func?

Vu Viet Phuong

@phuongvietvu0306

I am trying to use Basic auth on https://github.com/abourget/goproxy for HTTPS. Can anyone help me please? I cannot get it to work

Serhii Chaplia

@iwex

Hello! Guys how can I proxy proxied request ? :) I should change http.Transport for goproxy client

Lioxor64

@lioxor64

Hello, my name is Liora and I am currently hunting open source projects for company NeuraLegion (www.neuralegion.com)
We just launched a free annual subscription for open source projects for our AIAST tool NexPloit.

If you are interested, please, reach us on opensource@neuralegion.com!

Thank you for your time and consideration! If you have any questions, please do not hesitate to contact us!

Best,

Stoyan Kirov

@ctoyan

Hi everyone. I'm having troubles using the transparent example. I use all default settings and only change the GOPROXY_SERVER IP address.
The problem is that I'm not getting any requests to go to the proxy. No pages are loading, which means that the iptables rules are doing something, but they don't seem to route the requests to the proxy, because I get no logs (in verbose mode). When I make any requests, to any website, all I get is a failure to resolve the hostname.

Forgot to mention that I tried Docker and an Ubuntu virtual machine.
Has anyone had a similar problem?

Arun Kumar Pariyar

@openarun

Hi everyone, anyone around please ?

Bijay Singh

@HackProAIT_gitlab

getting too much trouble using this module can anyone help?

Vishnu V

@vvishnu_gitlab

Hi
I am trying to cache request using elazarl/goproxy. If request is not cached, it needs to be routed through an external proxy.

    proxy.OnRequest().DoFunc(func(req *http.Request,ctx *goproxy.ProxyCtx)(*http.Request,*http.Response) {
        if alreadyCached {
            return req, getCachedResponse(req)
        }
        //Here I want to route req via external proxy, I tried the following
        transport := &http.Transport{Proxy: http.ProxyURL(proxyUrl), TLSClientConfig: &tls.Config{InsecureSkipVerify: true},}
        proxtResponse, err := client.Do(proxyReq)
        //cache the response
        return req, response
    }

But some sites are not loading properly on using the client.Do method.
Any suggestion to route http.Request via proxy.

Alain Gilbert

@alaingilbert

anyone knows if it is possible to get the headers from the response of a CONNECT request ?

Atılhan Emre Dursunoğlu

@Thelta

I posted in mailing list, however i just see this gitter.
I have been getting "Cannot write TLS response header from mitm'd client: write tcp 127.0.0.1:{ProxyPort}->127.0.0.1:{RandomPort}: write: broken pipe" warnings in Mitm connections. Are these normal?
Thanks.

@zz

Hi, Team! I have multiple IP addresses on my servers. How can I do to use all addresses to rotate?

oshellhahaha

@oshellhahaha

0,0

k-p-ani

@k-p-ani

Hello Everyone, anyone knows how to proxy target endpoint which is a secured TLS endpoint using goproxy ? any pointers or reference is much appreciated

Where communities thrive

People

Repo info

Activity