Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Alex Walker
    @amlwwalker
    @abourget I am trying to allow sites, that I just can't MITM to just go through on the real SSL connection
    My project is open source, you can see my application here: https://github.com/DecodedCo/Mustard/blob/master/app/controllers/ProxyControllers.go
    on line 86 I am trying to allow connections that are SSL, that I havent been able to intercept (Im not installing a certificate on the client), through to the requested site

    So when someone loads a website, the page may be loaded over HTTPS and there is nothing I can do about it, but they might happen to load an HTTP page that I can inject into, I would like to allow anything through over HTTPS and only inject code into a page that is HTTP (or demonstrate how a picture loaded over HTTP is vulnerable etc etc. But otherwise the persons experience is exactly how it would be if there was no attacker (smooth as possible for the client)

    My app that I linked to above is the beginnings of a simple web based gui for controlling a MITM proxy

    Alex Walker
    @amlwwalker
    also is there a way to know when the current page is the / page so that if I inject into it, I dont inject into all HTML files, just that one
    Alexandre Bourget
    @abourget
    ok.. so on HandleConnect calls.. you might want to check the port the user is requesting..
    and if the port is 443.. then you'd know they probably want to connect via HTTPS.. then let it go through.
    but if it's port 80 or some other port.. you then return goproxy.MITM .. and then handle the query in HandleRequestFunc...
    FORWARD will not let you inspect the content.. it will merely close its eyes..
    so if you MITM an HTTP connection.. there's no SSL certs involved..
    it's possible some people use CONNECT even to go through port 80 ...
    Alexandre Bourget
    @abourget
    @amlwwalker ^^
    regarding the /.. see in my code up here ^^ .. there ctx.Req.URL.Path and such.. you can check the Host, the Path and all.. it's a plain Go http.Request
    once in the HandleRequestFunc section though.. you won't have the path in the HandleConnect callback.. as the Request there will be the CONNECT call..
    Alex Walker
    @amlwwalker

    @abourget so I am doing a goproxy.FORWARD:

     proxy.HandleConnectFunc( func(ctx *goproxy.ProxyCtx) goproxy.Next {
            // Potentially best to REJECT so that MITM is not detected. detection of MITM could cause suspicion
            if ctx.SNIHost() != "" {
                //log.Printintln(" *** HTTPS Connection: ", ctx.SNIHost())
                //return goproxy.MITM // This is failing...
                return goproxy.NEXT
            }
            return goproxy.FORWARD
        })

    But it feels like going inside this function is causing the SSL Error. Or should I just ALWAYS return a FORWARD?

    Nick Ma
    @nma
    Hi, just a quick question
    I'm trying to setup a transparent proxy in front of a docker instance
    for network / call / request metrics
    I set up 
    iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to 8888
    but I don't see any requests being proxied to goproxy, though for squid this is the accepted solution https://hub.docker.com/r/jpetazzo/squid-in-a-can/
    is goproxy the right tool for the job?
    shawokou123
    @shawokou123
    hi
    Elazar Leibovich
    @elazarl
    @nma I believe that it is possible to do with goproxy
    @nma search for transparent proxy in the goproxy-dev mailinglist
    Alexandre Bourget
    @abourget
    @elazarl so, what do we do with my fork ?
    Elazar Leibovich
    @elazarl
    @abourget I'll start the design ASAP
    I really hope that we'll have some shared doc for you guys to review by next Tuesday
    Almog Baku
    @AlmogBaku
    hi @all
    Almog Baku
    @AlmogBaku

    just wished to share with you with my stress test results compared to squid :smile: :
    Squid proxy:

    ubuntu@client:~$ ab -c 700 -n 40000 -X squid.proxy:3128 http://tapuz.co.il/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking tapuz.co.il [through squid.proxy:3128] (be patient)
Completed 4000 requests
Completed 8000 requests
Completed 12000 requests
Completed 16000 requests
Completed 20000 requests
Completed 24000 requests
Completed 28000 requests
Completed 32000 requests
Completed 36000 requests
Completed 40000 requests
Finished 40000 requests

Server Software:        
Server Hostname:        tapuz.co.il
Server Port:            80
Document Path:          /
Document Length:        233 bytes
Concurrency Level:      700
Time taken for tests:   20.550 seconds
Complete requests:      40000
Failed requests:        0
Non-2xx responses:      40000
Total transferred:      26200000 bytes
HTML transferred:       9320000 bytes
Requests per second:    1946.48 [#/sec] (mean)
Time per request:       359.623 [ms] (mean)
Time per request:       0.514 [ms] (mean, across all concurrent requests)
Transfer rate:          1245.07 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   45  19.7     45     122
Processing:   131  312  64.1    302    1058
Waiting:      131  312  64.0    301    1058
Total:        147  357  68.3    348    1089
Percentage of the requests served within a certain time (ms)
  50%    348
  66%    372
  75%    390
  80%    402
  90%    441
  95%    484
  98%    534
  99%    571
 100%   1089 (longest request)

    Goproxy:

    ubuntu@client:~$ ab -c 700 -n 40000 -X go.proxy:8080 http://tapuz.co.il/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking tapuz.co.il [through go.proxy:8080] (be patient)
Completed 4000 requests
Completed 8000 requests
Completed 12000 requests
Completed 16000 requests
Completed 20000 requests
Completed 24000 requests
Completed 28000 requests
Completed 32000 requests
Completed 36000 requests
Completed 40000 requests
Finished 40000 requests

Server Software:        
Server Hostname:        tapuz.co.il
Server Port:            80
Document Path:          /
Document Length:        55 bytes
Concurrency Level:      700
Time taken for tests:   9.727 seconds
Complete requests:      40000
Failed requests:        34668
   (Connect: 0, Receive: 0, Length: 34668, Exceptions: 0)
Non-2xx responses:      40000
Total transferred:      16361249 bytes
HTML transferred:       6819754 bytes
Requests per second:    4112.45 [#/sec] (mean)
Time per request:       170.215 [ms] (mean)
Time per request:       0.243 [ms] (mean, across all concurrent requests)
Transfer rate:          1642.69 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   22  78.5     15    1037
Processing:    10  145 106.1    163     665
Waiting:       10  145 106.2    163     665
Total:         17  167 132.4    180    1557
Percentage of the requests served within a certain time (ms)
  50%    180
  66%    214
  75%    238
  80%    253
  90%    294
  95%    338
  98%    411
  99%    475
 100%   1557 (longest request)
    tested on the same ec2 flavor
    cc: @elazarl
    Elazar Leibovich
    @elazarl

    Thanks @AlmogBaku , Thanks. I'm glad the longest request is only 50% longer than squid.

    I'd consider using https://github.com/giltene/wrk2 for a better latency analysis by Gil Tenne

    I'll try to mention it in the wiki
    Almog Baku
    @AlmogBaku
    @elazarl on the previous test it was faster... BUT unfortunately it seems like my test was wrong

    the goproxy had lots of missed requests.. im seeing from the console this:

    2016/03/09 12:31:51 [086] WARN: Error copying to client: read tcp 172.31.25.102:8080->198.27.114.146:34600: read tcp 172.31.25.102:49891->204.79.197.200:80: read: connection reset by peer
2016/03/09 12:31:52 [085] WARN: Error copying to client: read tcp 172.31.25.102:8080->198.27.114.146:34594: read tcp 172.31.25.102:49890->204.79.197.200:80: read: connection reset by peer

    and

    2016/03/09 12:32:29 [118] WARN: Error copying to client: read tcp 172.31.25.102:8080->158.69.118.48:50948: write tcp 172.31.25.102:8080->158.69.118.48:50948: use of closed network connection
2016/03/09 12:32:29 [117] WARN: Error copying to client: read tcp 172.31.25.102:8080->158.69.118.48:50646: write tcp 172.31.25.102:8080->158.69.118.48:50646: use of closed network connection
    and
    2016/03/09 13:12:01 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms
2016/03/09 13:12:01 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms
2016/03/09 13:12:01 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 10ms
2016/03/09 13:12:02 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms
2016/03/09 13:12:02 http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms

    I executed the goproxy from root, and exectue ulimit -n 10240 before :|

    open files while the stress test is around 1024:

    root@ip-172-31-25-102:/home/ubuntu# lsof -p 3946|wc -l
1024

    open files after is 13:
    root@ip-172-31-25-102:/home/ubuntu# lsof -p 3946|wc -l 13

    limit:

    root@ip-172-31-25-102:/home/ubuntu# ulimit -n
10240
    Almog Baku
    @AlmogBaku
    ahhhh... i need to run the ulimt from the same tty of the goproxy!
    now it worked.. and the results are still better than squid.. but i still had few errors of the tcp write... 
    untu@ip-172-31-25-103:~$ ab -c 700 -n 40000 -X go.proxy:8080 http://tapuz.co.il/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking tapuz.co.il [through go.proxy:8080] (be patient)
Completed 4000 requests
Completed 8000 requests
Completed 12000 requests
Completed 16000 requests
Completed 20000 requests
Completed 24000 requests
Completed 28000 requests
Completed 32000 requests
Completed 36000 requests
Completed 40000 requests
Finished 40000 requests


Server Software:        
Server Hostname:        tapuz.co.il
Server Port:            80

Document Path:          /
Document Length:        233 bytes

Concurrency Level:      700
Time taken for tests:   19.765 seconds
Complete requests:      40000
Failed requests:        0
Non-2xx responses:      40000
Total transferred:      20600000 bytes
HTML transferred:       9320000 bytes
Requests per second:    2023.76 [#/sec] (mean)
Time per request:       345.892 [ms] (mean)
Time per request:       0.494 [ms] (mean, across all concurrent requests)
Transfer rate:          1017.81 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   16  15.1     12      70
Processing:   107  325 124.5    299    1100
Waiting:      107  325 124.5    299    1100
Total:        107  341 128.0    315    1117

Percentage of the requests served within a certain time (ms)
  50%    315
  66%    386
  75%    429
  80%    457
  90%    520
  95%    574
  98%    636
  99%    681
 100%   1117 (longest request)
    only those three errors
    root@ip-172-31-25-102:/home/ubuntu# ./goproxy-basic 
2016/03/09 13:28:29 [180] WARN: Error copying to client: read tcp 172.31.25.102:8080->188.68.224.51:32895: write tcp 172.31.25.102:8080->188.68.224.51:32895: use of closed network connection
2016/03/09 13:29:34 [073] WARN: Error copying to client: read tcp 172.31.25.102:48632->212.74.50.1:443: write tcp 172.31.25.102:48632->212.74.50.1:443: use of closed network connection
2016/03/09 13:31:08 [081] WARN: Error copying to client: read tcp 172.31.25.102:8080->188.68.224.51:43394: write tcp 172.31.25.102:8080->188.68.224.51:43394: use of closed network connection
    Almog Baku
    @AlmogBaku
    @elazarl wrk2 doesn't support proxy out of the box...
    Almog Baku
    @AlmogBaku
    @elazarl do you have any idea regarding: elazarl/goproxy#149 ?
    Almog Baku
    @AlmogBaku
    see golang/go#15015
    Alex Scotton
    @Illizian
    Hi, please excuse my ignorance. This is my first foray into Golang. I am trying to write a custom ReqCondition func. I unfortunately lack the knowledge to understand why it doesn't work.
    Here is the prototype func: 
    func isBlacklisted() goproxy.ReqCondition {
  return goproxy.ReqConditionFunc(func(req *http.Request, ctx *goproxy.ProxyCtx) bool {
    return true;
  })
}
    and I am including thusly: 
    proxy.OnRequest(isBlacklisted()).HandleConnect(goproxy.AlwaysReject)
    the isBlacklisted func is being called but the ReqConditionFunc is not, I have also tried that without the wrapper
    Alex Scotton
    @Illizian
    scrap that! the issue this whole time was my use of goproxy.AlwaysReject , Pretty sure I'm doing it wrong :P as I'm already useing HandleConnect with an AlwaysMitM function, can you only have one HandleConnect statement?