Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Aug 20 21:17

    finaglehelper on develop

    Remove console formatting Prob… (compare)

  • Aug 20 18:34

    finaglehelper on develop

    inject-server: Update c.t.injec… (compare)

  • Aug 20 08:05
    szysas edited #507
  • Aug 20 07:33
    szysas opened #507
  • Aug 19 21:17

    finaglehelper on develop

    Ensure read from files as bytes… (compare)

  • Aug 19 18:17

    finaglehelper on develop

    finatra: Add test for HttpRespo… (compare)

  • Aug 15 22:33
    kevinoliver labeled #504
  • Aug 15 22:24
    kevinoliver labeled #505
  • Aug 15 21:16

    finaglehelper on develop

    Twitter-oss: update OSS librari… (compare)

  • Aug 15 18:24

    finaglehelper on develop

    Twitter-oss: Prepare OSS util f… (compare)

  • Aug 15 17:17

    finaglehelper on develop

    Move JVM flags from the sbt scr… (compare)

  • Aug 14 21:50
    codecov[bot] commented #506
  • Aug 14 21:49
    codecov[bot] commented #506
  • Aug 14 20:58
    CLAassistant commented #506
  • Aug 14 20:57
    bryce-anderson opened #506
  • Aug 14 16:39
    codecov[bot] commented #505
  • Aug 14 16:08
    CLAassistant commented #505
  • Aug 14 16:07
    CLAassistant commented #505
  • Aug 14 16:07
    guillaume-chech opened #505
  • Aug 12 22:40

    finaglehelper on develop

    finatra-kafka: adding additiona… (compare)

Christopher Coco
@cacoco
:thumbsup:
sinanspd
@sinanspd
@cacoco is there a suggested way to do Request level global processing/error handling? I tried the following (it has some type error atm) but it seems like an overkill. Finch had a RequestMonad that had this idea, not sure about finatra. The goal is that we have all these domain objects that will throw very similar errors like NotFound, DublicateKey etc. so I am curious if we can have all the controllers call one function in stead of repeatedly doing try catch blocks for every end point
               commonRequestHandler[T, R, M <: {def toResponse(entity: T): Response}](f: () => T, request: R, mapper: M): Response = {
                      try{
                               responseMapper.toResponse(callback())
                      }catch{
                            // Catch all the standard errors here and convert to corresponding response 
                      } 
             }
Christopher Coco
@cacoco
@sinanspd did you take a look at the user guide section on exception mapping which goes into detail about this? https://twitter.github.io/finatra/user-guide/http/exceptions.html
sinanspd
@sinanspd
aaa I realized I was using the mappers wrong. I was injecting them into the services, didn't realize I can inject them into the server directly... My bad. Thanks!
Christopher Coco
@cacoco
You don't inject them per se
The key is the Exception Mapping filter which translates things in the response path of the server that is on the way out and back to the caller
This filter if it encounters an exception looks up in the exception manager if there is a mapper that will translate the exception into a prope me httpnl reponse
You would never need to use a mapper directly
So all you need to do is throw proper exceptions and you can define a way for the server to translate those exceptions to responses.
Hope that helps
sinanspd
@sinanspd
Makes perfect sense. I over dramatized the situation as usual. Thanks for the info!
genesis-root
@genesis-root
any suggestions what to use for local load testing?
alice
@agutteridge

Hello! I was wondering if there are any general recommendations for configuring a Finagle Client + Finatra Server combination to ensure that the client doesn't overwhelm the server. In our use case, the client is part of a scheduled job that sends 100s of thousands of requests, however the server needs to be able to respond to other users as well. The server was originally Spring-based but seems to be causing more problems now that we've converted it to Finatra.

We've made the connection pool smaller on the client side, but this is resulting in service creation failures after the job has been running for some time. We expected connection requests to be queued rather than seeing these errors - is there anything obvious we might be missing?

Christopher Coco
@cacoco
Hi there, would need more information (like what errors you're seeing if any or the pattern of behavior exhibited or specifics about what your server metrics looked like) but have you looked at the Finagle client and server documentation? Also, are you blocking the main thread by any chance? https://twitter.github.io/finagle/guide/Clients.html and https://twitter.github.io/finagle/guide/Servers.html and https://twitter.github.io/finagle/guide/Metrics.html and https://finagle.github.io/blog/2016/09/01/block-party/
alice
@agutteridge
Hi Christopher, thanks for the reply! Yes we've gone through the docs, and are using Futures all the way through. The specific error we're seeing is exceeded 2.seconds to <service> while creating a service/connection or reserving a service/connection from the service/connection pool in the Finagle client.
sinanspd
@sinanspd
@genesis-root a little bit late but you can do something like this with FeatureTests
test(....){
        pending
        val testPool = FuturePools.fixedPool("...", size = 10000)
        for(i <- 1 to 100){
            testPool{
                    server.httpGet(path="....", andExpect=Status.Ok)
            }
        }
        Thread.sleep(100)
}
feel free to play around with the params
Christopher Coco
@cacoco
@agutteridge you’re seeing this error in the finagle client? My guess is that you are starving the client’s threads in some manner. Perhaps by not cleaning up properly.
@sinanspd I would never recommend to do a Thread.sleep in tests. @genesis-root running a “load test” on a single machine locally is likely to not really tell you anything. You most likely want to perform benchmarking (microbenchmarking) with something like JMH. For true load testing you’d want a replica of your production environment and load generated most likely from many machines to simulate more than what a single box can generate.
@agutteridge can you share how you create and configure your client?
Christopher Coco
@cacoco
I would also take a look at the client pooling documentation: https://twitter.github.io/finagle/guide/Clients.html#pooling
@genesis-root if you really want something you can run “locally”, the apache bench (ab) tool can be useful: http://httpd.apache.org/docs/current/programs/ab.html
alice
@agutteridge
@cacoco thanks for the tip, we'll investigate and then get back to you :thumbsup:
Patrick OBrien
@pobrienAvlino
Seeing this error when trying to integrate with Kafka and using with QueryableState
StaticPartitioning currently requires flag 'kafka.application.server' to be set to the Aurora service proxy hostname e.g. 0.tweet-word-count.prod.team1.service.smf1.twitter.com:12345
Can this be used with out Aurora?
Christopher Coco
@cacoco
Pretty sure that can be wherever the Kafka application is running addressable via a host and port.
Patrick OBrien
@pobrienAvlino
@cacoco Thanks for responding. What is meant by Kafka application? Do you mean the broker, zookeeper ? Something else entirely.
jdcohen220
@jdcohen220
Is it possible to set the initial offset for a KafkaStreamsTwitterServer to latest?
Steve Cosenza
@scosenza
@pobrienAvlino To use FinatraKafkaStream's StaticPartitioning trait without Aurora, you can set "kafka.application.server" to be "<instance_id>.:<port>". Make sure the <instance_id> matches the value in the "kafka.current.shard" flag. The <port> isn't used but must be included. The port can be any value greater than 0. For example, to represent shard 5, you could specify kafka.application.server=5:1
mingaliu
@mingaliu
@jdcohen220 , you can use the process flag of "-kafka.consumer.auto.offset.reset=latest", which means to start from latest if there is no initial offset. If you want to start from the latest with existing offset, you can use the Stream Reset tool, https://cwiki.apache.org/confluence/display/KAFKA/KIP-171+-+Extend+Consumer+Group+Reset+Offset+for+Stream+Application
Andrew Snowden
@andrewsnowden
Hi guys - I'm having a lot of issues with HttpClient failing and never recovering. I feel like I must be doing something completely wrong - I was assuming that I make one Http.client and then re-use it for multiple requests - is that right? (I'm getting a ChannelClosedException but I'm not seeing any HTTP traffic on the server it is supposed to be hitting)
We've tried setting a bunch of different FailFast and FailureAccrual settings but these don't seem to have helped us
Patrick OBrien
@pobrienAvlino
Is there a new example of using the QueryableState? I have gotten through my initial exceptions, but the examples here https://twitter.github.io/finatra/user-guide/kafka-streams/index.html#queryable-state and here https://twitter.github.io/finatra/user-guide/kafka-streams/examples.html#queryable-state don't match the source code I am looking through. While I am able to create the streams correctly and as far as I can tell the stores are created as well. The stores show up in the topology, and I can see them in the admin page. However when I try to access these stores via queryableFinatraKeyValueStore, the stores are not found. I can't even find the stores in FinatraStoresGlobalManager.getStores ... I was unable to locate in the source code where these stores would be put into the FinatraStoresGlobalManager object.
Steve Cosenza
@scosenza
@pobrienAvlino Our apologies that the QueryableState example isn't up to date! The Finatra QueryableState trait currently can only be used when using a FinatraTransformer (which adds some functionality on top of Kafka's Transformer class). The example shown here: https://twitter.github.io/finatra/user-guide/kafka-streams/examples.html#queryable-state is using ".count()(Materialized.as(countStoreName))" which means a normal Kafka's transformer is being used. A correct example would have used FinatraDslWindowedAggregations#sum instead
Steve Cosenza
@scosenza

@pobrienAvlino To implement "Queryable State" with the built in Kafka aggregations and state stores, you can use the KafkaStreams class as such e.g.

Note: Typing the following on my phone so won't likely compile :-)

final override def configureThrift(router: ThriftRouter): Unit = {
    router
      .add(
        new WordCountQueryService(
        kafkaStreams
  .store(countStoreName)
  .asInstanceOf[ReadOnlyKeyValueStore[String, Long]]
        )
      )
  }
class WordCountQueryService(
  //Note: Make sure ReadOnlyKeyValueStore is "pass by name"
  store: => ReadOnlyKeyValueStore[String, Long]]) {
  ...
}
Vianel
@vianel

Hello, has anyone had this issue? I have these case classes

case class Item(priority: Double) case class Response(serverTime: Long, timeZoneOffset: Int, data: Item)
so the response of my endpoint looks like this

{
    "serverTime": "1565292753751",
    "timeZoneOffset": "-18000000",
    "data": {
        "priority": "18.3333"
    }
}

why the priority field is a string even when I declared as a double?

Vianel
@vianel
I figured out :D
FinatraJacksonModule made the conversion
Christopher Coco
@cacoco
Yes pretty sure the default configuration is to render numbers as strings otherwise you are very likely to lose precision with JavaScript representation. This is generally considered good practice, fwiw.
Martinffx
@martinffx
Hi, I'm having an issue with the MDC Logging. If I log in a controller that has MDC setup via filter it includes the value in the log message, as soon as I log from within an injected class MDC is lost from the log message.
I thought the MDC logging was supposed to transparently move across threads via Contexts... what am I missing do I need to configure something for the value to flow outside of the controlled context?
@Singleton
class MyController @Inject()(myService: MyService) extends Controller {
  post("/test") { rq: MyRequest =>
    info("controller")
    myService.test(rq)
  }
}
Martinffx
@martinffx
import com.twitter.inject.Logging

@Singleton
class MyService @Inject()(val repo: Repository) extends Logging {
  def test(rq: MyRequest): Future[MyResponse] = {
    info("service")
  }
}
will give me:
Christopher Coco
@cacoco
@martinffx if you log from an injected class, that needs to be happening within the scope of an initialized MDC context (as described here: https://twitter.github.io/finatra/user-guide/logging/mdc.html#usage). If you are calling the MDC in the injected class this still needs to happen in the scope of an outer initialization. If you are doing that and it is not working, it would be great to file an issue.
Martinffx
@martinffx
I'm going to try to create an isolated example that reproduces the issue
I'm also having issues with the Client, it is constantly marking itself as dead and not coming back in a timely manner.
is there a recommended config for low volume servers?
Christopher Coco
@cacoco
@martinffx there are recommendations if your client to is pointing so a single host (like a VIP). You would disable failfast and failure accrual. See "circuit breaking" here: https://twitter.github.io/finagle/guide/Clients.html