Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Nick Ma
    @nma
    but I don't see any requests being proxied to goproxy, though for squid this is the accepted solution https://hub.docker.com/r/jpetazzo/squid-in-a-can/
    is goproxy the right tool for the job?
    shawokou123
    @shawokou123
    hi
    Elazar Leibovich
    @elazarl
    @nma I believe that it is possible to do with goproxy
    @nma search for transparent proxy in the goproxy-dev mailinglist
    Alexandre Bourget
    @abourget
    @elazarl so, what do we do with my fork ?
    Elazar Leibovich
    @elazarl
    @abourget I'll start the design ASAP
    I really hope that we'll have some shared doc for you guys to review by next Tuesday
    Almog Baku
    @AlmogBaku
    hi @all
    Almog Baku
    @AlmogBaku

    just wished to share with you with my stress test results compared to squid :smile: :
    Squid proxy:

    ubuntu@client:~$ ab -c 700 -n 40000 -X squid.proxy:3128 http://tapuz.co.il/
    This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/
    Benchmarking tapuz.co.il [through squid.proxy:3128] (be patient)
    Completed 4000 requests
    Completed 8000 requests
    Completed 12000 requests
    Completed 16000 requests
    Completed 20000 requests
    Completed 24000 requests
    Completed 28000 requests
    Completed 32000 requests
    Completed 36000 requests
    Completed 40000 requests
    Finished 40000 requests
    
    Server Software:        
    Server Hostname:        tapuz.co.il
    Server Port:            80
    Document Path:          /
    Document Length:        233 bytes
    Concurrency Level:      700
    Time taken for tests:   20.550 seconds
    Complete requests:      40000
    Failed requests:        0
    Non-2xx responses:      40000
    Total transferred:      26200000 bytes
    HTML transferred:       9320000 bytes
    Requests per second:    1946.48 [#/sec] (mean)
    Time per request:       359.623 [ms] (mean)
    Time per request:       0.514 [ms] (mean, across all concurrent requests)
    Transfer rate:          1245.07 [Kbytes/sec] received
    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0   45  19.7     45     122
    Processing:   131  312  64.1    302    1058
    Waiting:      131  312  64.0    301    1058
    Total:        147  357  68.3    348    1089
    Percentage of the requests served within a certain time (ms)
      50%    348
      66%    372
      75%    390
      80%    402
      90%    441
      95%    484
      98%    534
      99%    571
     100%   1089 (longest request)

    Goproxy:

    ubuntu@client:~$ ab -c 700 -n 40000 -X go.proxy:8080 http://tapuz.co.il/
    This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/
    Benchmarking tapuz.co.il [through go.proxy:8080] (be patient)
    Completed 4000 requests
    Completed 8000 requests
    Completed 12000 requests
    Completed 16000 requests
    Completed 20000 requests
    Completed 24000 requests
    Completed 28000 requests
    Completed 32000 requests
    Completed 36000 requests
    Completed 40000 requests
    Finished 40000 requests
    
    Server Software:        
    Server Hostname:        tapuz.co.il
    Server Port:            80
    Document Path:          /
    Document Length:        55 bytes
    Concurrency Level:      700
    Time taken for tests:   9.727 seconds
    Complete requests:      40000
    Failed requests:        34668
       (Connect: 0, Receive: 0, Length: 34668, Exceptions: 0)
    Non-2xx responses:      40000
    Total transferred:      16361249 bytes
    HTML transferred:       6819754 bytes
    Requests per second:    4112.45 [#/sec] (mean)
    Time per request:       170.215 [ms] (mean)
    Time per request:       0.243 [ms] (mean, across all concurrent requests)
    Transfer rate:          1642.69 [Kbytes/sec] received
    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0   22  78.5     15    1037
    Processing:    10  145 106.1    163     665
    Waiting:       10  145 106.2    163     665
    Total:         17  167 132.4    180    1557
    Percentage of the requests served within a certain time (ms)
      50%    180
      66%    214
      75%    238
      80%    253
      90%    294
      95%    338
      98%    411
      99%    475
     100%   1557 (longest request)
    tested on the same ec2 flavor
    cc: @elazarl
    Elazar Leibovich
    @elazarl

    Thanks @AlmogBaku , Thanks. I'm glad the longest request is only 50% longer than squid.

    I'd consider using https://github.com/giltene/wrk2 for a better latency analysis by Gil Tenne

    I'll try to mention it in the wiki
    Almog Baku
    @AlmogBaku
    @elazarl on the previous test it was faster... BUT unfortunately it seems like my test was wrong

    the goproxy had lots of missed requests.. im seeing from the console this:

    2016/03/09 12:31:51 [086] WARN: Error copying to client: read tcp 172.31.25.102:8080->198.27.114.146:34600: read tcp 172.31.25.102:49891->204.79.197.200:80: read: connection reset by peer
    2016/03/09 12:31:52 [085] WARN: Error copying to client: read tcp 172.31.25.102:8080->198.27.114.146:34594: read tcp 172.31.25.102:49890->204.79.197.200:80: read: connection reset by peer

    and

    2016/03/09 12:32:29 [118] WARN: Error copying to client: read tcp 172.31.25.102:8080->158.69.118.48:50948: write tcp 172.31.25.102:8080->158.69.118.48:50948: use of closed network connection
    2016/03/09 12:32:29 [117] WARN: Error copying to client: read tcp 172.31.25.102:8080->158.69.118.48:50646: write tcp 172.31.25.102:8080->158.69.118.48:50646: use of closed network connection
    and
    2016/03/09 13:12:01 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms
    2016/03/09 13:12:01 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms
    2016/03/09 13:12:01 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 10ms
    2016/03/09 13:12:02 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms
    2016/03/09 13:12:02 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms

    I executed the goproxy from root, and exectue ulimit -n 10240 before :|

    open files while the stress test is around 1024:

    root@ip-172-31-25-102:/home/ubuntu# lsof -p 3946|wc -l
    1024

    open files after is 13:
    root@ip-172-31-25-102:/home/ubuntu# lsof -p 3946|wc -l 13

    limit:

    root@ip-172-31-25-102:/home/ubuntu# ulimit -n
    10240
    Almog Baku
    @AlmogBaku
    ahhhh... i need to run the ulimt from the same tty of the goproxy!
    now it worked.. and the results are still better than squid.. but i still had few errors of the tcp write...
    untu@ip-172-31-25-103:~$ ab -c 700 -n 40000 -X go.proxy:8080 http://tapuz.co.il/
    This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/
    
    Benchmarking tapuz.co.il [through go.proxy:8080] (be patient)
    Completed 4000 requests
    Completed 8000 requests
    Completed 12000 requests
    Completed 16000 requests
    Completed 20000 requests
    Completed 24000 requests
    Completed 28000 requests
    Completed 32000 requests
    Completed 36000 requests
    Completed 40000 requests
    Finished 40000 requests
    
    
    Server Software:        
    Server Hostname:        tapuz.co.il
    Server Port:            80
    
    Document Path:          /
    Document Length:        233 bytes
    
    Concurrency Level:      700
    Time taken for tests:   19.765 seconds
    Complete requests:      40000
    Failed requests:        0
    Non-2xx responses:      40000
    Total transferred:      20600000 bytes
    HTML transferred:       9320000 bytes
    Requests per second:    2023.76 [#/sec] (mean)
    Time per request:       345.892 [ms] (mean)
    Time per request:       0.494 [ms] (mean, across all concurrent requests)
    Transfer rate:          1017.81 [Kbytes/sec] received
    
    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0   16  15.1     12      70
    Processing:   107  325 124.5    299    1100
    Waiting:      107  325 124.5    299    1100
    Total:        107  341 128.0    315    1117
    
    Percentage of the requests served within a certain time (ms)
      50%    315
      66%    386
      75%    429
      80%    457
      90%    520
      95%    574
      98%    636
      99%    681
     100%   1117 (longest request)
    only those three errors
    root@ip-172-31-25-102:/home/ubuntu# ./goproxy-basic 
    2016/03/09 13:28:29 [180] WARN: Error copying to client: read tcp 172.31.25.102:8080->188.68.224.51:32895: write tcp 172.31.25.102:8080->188.68.224.51:32895: use of closed network connection
    2016/03/09 13:29:34 [073] WARN: Error copying to client: read tcp 172.31.25.102:48632->212.74.50.1:443: write tcp 172.31.25.102:48632->212.74.50.1:443: use of closed network connection
    2016/03/09 13:31:08 [081] WARN: Error copying to client: read tcp 172.31.25.102:8080->188.68.224.51:43394: write tcp 172.31.25.102:8080->188.68.224.51:43394: use of closed network connection
    Almog Baku
    @AlmogBaku
    @elazarl wrk2 doesn't support proxy out of the box...
    Almog Baku
    @AlmogBaku
    @elazarl do you have any idea regarding: elazarl/goproxy#149 ?
    Almog Baku
    @AlmogBaku
    see golang/go#15015
    Alex Scotton
    @Illizian
    Hi, please excuse my ignorance. This is my first foray into Golang. I am trying to write a custom ReqCondition func. I unfortunately lack the knowledge to understand why it doesn't work.
    Here is the prototype func:
    func isBlacklisted() goproxy.ReqCondition {
      return goproxy.ReqConditionFunc(func(req *http.Request, ctx *goproxy.ProxyCtx) bool {
        return true;
      })
    }
    and I am including thusly:
    proxy.OnRequest(isBlacklisted()).HandleConnect(goproxy.AlwaysReject)
    the isBlacklisted func is being called but the ReqConditionFunc is not, I have also tried that without the wrapper
    Alex Scotton
    @Illizian
    scrap that! the issue this whole time was my use of goproxy.AlwaysReject , Pretty sure I'm doing it wrong :P as I'm already useing HandleConnect with an AlwaysMitM function, can you only have one HandleConnect statement?
    ZiYuan
    @Bpazy
    Hi, I have a problem:
    func handler(r *http.Request, ctx *goproxy.ProxyCtx) (*http.Request, *http.Response) {
        temp := make([]byte, 1024)
        r.Body.Read(temp) // How to seek to start?
        return r, nil
    }
    Sriharsha Setty
    @horsey
    Hello!
    Sriharsha Setty
    @horsey
    @elazarl I came across goproxy while looking around for a CORS proxy server. Basically I am looking at a way to proxy requests for servers that don't handle the CORS very well.
    Sriharsha Setty
    @horsey
    Can I setup goproxy to accept requests in a manner: www.goproxy.com/http://api-server.com to proxy the request to http://api-server.com?
    GNU
    @cuu
    hello all, I just wonder if anyone has asked, how to get the raw request strings from this goproxy ? espcially in HandleConnectFunc with CONNECT MOTHOD...
    Odohi David
    @spankie
    :point_up: December 7, 2016 8:49 PM this proxy will help you with your CORS issue.
    ilk
    @ilk
    Hi - How can access the https response body with goproxy? I have some jobs which fetch xml and json (written in different langs) and they should use a central proxy which save all the responses. I need that for logging/history purposes.
    Mohamed Isoukrane
    @misoukrane
    Hi, I am trying to proxy HTTPS through connect without using MITM, all what I see in the transparent example is AlwaysMitm. is there a different option that I can use to achieve proxying HTTPS without mitm? Thanks in advance.
    tsjnsn
    @tsjnsn
    Do I need to do anything to enable keep alive?
    tsjnsn
    @tsjnsn
    Guess so
    func responseIsKeepAlive(res *http.Response) bool {
          // TODO: implement.  for now just always shutting down the connection.
    SuperFashi
    @hanbang-wang
    Hi, one question. Why some of the requests are invisible in OnRequest func?
    Vu Viet Phuong
    @phuongvietvu0306
    I am trying to use Basic auth on https://github.com/abourget/goproxy for HTTPS. Can anyone help me please? I cannot get it to work
    Serhii Chaplia
    @iwex
    Hello! Guys how can I proxy proxied request ? :) I should change http.Transport for goproxy client
    Lioxor64
    @lioxor64

    Hello, my name is Liora and I am currently hunting open source projects for company NeuraLegion (www.neuralegion.com)
    We just launched a free annual subscription for open source projects for our AIAST tool NexPloit.

    If you are interested, please, reach us on opensource@neuralegion.com!

    Thank you for your time and consideration! If you have any questions, please do not hesitate to contact us!

    Best,

    Stoyan Kirov
    @ctoyan
    Hi everyone. I'm having troubles using the transparent example. I use all default settings and only change the GOPROXY_SERVER IP address.
    The problem is that I'm not getting any requests to go to the proxy. No pages are loading, which means that the iptables rules are doing something, but they don't seem to route the requests to the proxy, because I get no logs (in verbose mode). When I make any requests, to any website, all I get is a failure to resolve the hostname.
    Forgot to mention that I tried Docker and an Ubuntu virtual machine.
    Has anyone had a similar problem?
    Arun Kumar Pariyar
    @openarun
    Hi everyone, anyone around please ?
    Bijay Singh
    @HackProAIT_gitlab
    getting too much trouble using this module can anyone help?
    Vishnu V
    @vvishnu_gitlab

    Hi
    I am trying to cache request using elazarl/goproxy. If request is not cached, it needs to be routed through an external proxy.

        proxy.OnRequest().DoFunc(func(req *http.Request,ctx *goproxy.ProxyCtx)(*http.Request,*http.Response) {
            if alreadyCached {
                return req, getCachedResponse(req)
            }
            //Here I want to route req via external proxy, I tried the following
            transport := &http.Transport{Proxy: http.ProxyURL(proxyUrl), TLSClientConfig: &tls.Config{InsecureSkipVerify: true},}
            proxtResponse, err := client.Do(proxyReq)
            //cache the response
            return req, response
        }

    But some sites are not loading properly on using the client.Do method.
    Any suggestion to route http.Request via proxy.