oat++ framework/Lobby

oat++

@oatpp_io_twitter

Hey @AlexandreHURDYK , can you share your thoughts about Phytec boards with @acidtonic ?

mmn

@mmn:matrix.org

What is the canonical way to check whether a DTO field was "filled" during a deserialisation (using oatpp::parser::json::mapping::ObjectMapper)?

Or alternatively, is there a canonical way to make the deserialisation fail if not all required keys were present in the deserialised JSON (and, for example, throw an exception)?

There is the allowUnknownFields option for unknown fields, but I somehow need to handle the opposite (I need all known fields to be filled and fail if they are not).

mmn

@mmn:matrix.org

[m]

And finally, is there some recommended way to perform a partial deserialisation? Let's say that I have a json object with one value determining the nature of the rest of its contents, so first I need to extract that one value only, based on that value select the correct DTO, and only after that deserialise the whole object with the correct DTO.

junyama

@junyama

I tried to make MySecurApiCall Automatic Retries by following the doc:

OATPP_CREATE_COMPONENT(std::shared_ptr<MySecureApiCall>, mySecureApiClient)([] {
        OATPP_COMPONENT(std::shared_ptr<oatpp::network::ClientConnectionProvider>, connectionProvider, "clientConnectionProvider");
        OATPP_COMPONENT(std::shared_ptr<oatpp::data::mapping::ObjectMapper>, objectMapper);

        // create connection pool
        auto connectionPool = oatpp::network::ClientConnectionPool::createShared(
        connectionProvider, // connection provider
        10, //max connections
        std::chrono::seconds(5) // max lifetime of idle connection
        );
        // create retry policy
        auto retryPolicy = oatpp::web::client::SimpleRetryPolicy(2 /* max retries */, std::chrono::seconds(1) /* retry interval */);

        //auto requestExecutor = oatpp::web::client::HttpRequestExecutor::createShared(connectionProvider);
        auto requestExecutor = oatpp::web::client::HttpRequestExecutor::createShared(connectionPool, retryPolicy /* retry policy */);
        return MySecureApiCall::createShared(requestExecutor, objectMapper);
    }());

I have modified some lines of the example, but still have "no matching function for call compile" error for "auto requestExecutor = oatpp.... " line. Please tell me what was wrong. Thank you for your help in advance.

oat++

@oatpp_io_twitter

Hey @junyama ,
retryPolicy should be shared_ptr - https://github.com/oatpp/oatpp/blob/master/src/oatpp/web/client/HttpRequestExecutor.hpp#L115

Hello @mmn:matrix.org ,

What is the canonical way to check whether a DTO field was "filled" during a deserialisation (using oatpp::parser::json::mapping::ObjectMapper)?

At the moment it's the following:

ENDPOINT("POST", "users", createUser,
         BODY_DTO(Object<UserDto>, userDto))
{

  OATPP_ASSERT_HTTP(userDto->id, Status::CODE_400, "id field is required") 
  OATPP_ASSERT_HTTP(userDto->role, Status::CODE_400, "role field is required") 

  ...

}

Or, you can move those checks ex.: to UserDto:

class UserDto : oatpp::DTO {

...

  validate() {
    OATPP_ASSERT_HTTP(userDto->id, Status::CODE_400, "id field is required") 
    OATPP_ASSERT_HTTP(userDto->role, Status::CODE_400, "role field is required") 
  }

};

...

ENDPOINT("POST", "users", createUser,
         BODY_DTO(Object<UserDto>, userDto))
{

  userDto->validate(); 

  ...

}

2 replies

junyama

@junyama

Thank you for the quick response. I have modified my code as follows:

OATPP_CREATE_COMPONENT(std::shared_ptr<MySecureApiCall>, mySecureApiClient)([] {
        OATPP_COMPONENT(std::shared_ptr<oatpp::network::ClientConnectionProvider>, connectionProvider, "clientConnectionProvider");
        OATPP_COMPONENT(std::shared_ptr<oatpp::data::mapping::ObjectMapper>, objectMapper);

        // create connection pool
        auto connectionPool = std::make_shared<oatpp::network::ClientConnectionPool>(
        connectionProvider, // connection provider
        10, //max connections
        std::chrono::seconds(5) // max lifetime of idle connection
        );
        // create retry policy
        //auto retryPolicy = oatpp::web::client::SimpleRetryPolicy(2 /* max retries */, std::chrono::seconds(1) /* retry interval */);
        auto retryPolicy = std::make_shared<oatpp::web::client::SimpleRetryPolicy>(2 /* max retries */, std::chrono::seconds(1) /* retry interval */);


        //auto requestExecutor = oatpp::web::client::HttpRequestExecutor::createShared(connectionProvider);
        auto requestExecutor = oatpp::web::client::HttpRequestExecutor::createShared(connectionPool, retryPolicy /* retry policy */);
        return MySecureApiCall::createShared(requestExecutor, objectMapper);
    }());

but still showed the following compile error:

AppComponent.hpp:145:9:   required from here
/usr/include/c++/7/ext/new_allocator.h:136:4: error: no matching function for call to ‘oatpp::provider::Pool<oatpp::network::ClientConnectionProvider, oatpp::data::stream::IOStream, oatpp::network::ConnectionAcquisitionProxy>::Pool(std::shared_ptr<oatpp::network::ClientConnectionProvider>&, int, std::chrono::duration<long int>)’
  { ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

oat++

@oatpp_io_twitter

And finally, is there some recommended way to perform a partial deserialisation? Let's say that I have a json object with one value determining the nature of the rest of its contents, so first I need to extract that one value only, based on that value select the correct DTO, and only after that deserialise the whole object with the correct DTO.

You can use Polymorphic DTO fields for that - https://github.com/oatpp/oatpp/blob/master/changelog/1.3.0.md#polymorphic-dto_field

class MyDto : public oatpp::DTO {

  DTO_INIT(MyDto, DTO)

  DTO_FIELD(String, dataType); //<-- value determining the nature of the rest of its contents
  DTO_FIELD(Any, data); //<-- polymorphic data

  DTO_FIELD_TYPE_SELECTOR(data) {

    if(dataType == "animal") return Object<AnimalDto>::Class::getType();
    if(dataType == "car") return Object<CarDto>::Class::getType();
    if(dataType == "int") return Int32::Class::getType();

    return Void::Class::getType();
  }

};

@mmn:matrix.org

1 reply

@junyama

    auto connectionPool = oatpp::network::ClientConnectionPool::createShared(
      connectionProvider, // connection provider
      10, //max connections
      std::chrono::seconds(5) // max lifetime of idle connection
    );

use ClientConnectionPool::createShared instead

junyama

@junyama

No compile error. Thank you.

mmn

@mmn:matrix.org

[m]

So based on the value of dataType, I would need to select a whole DTO rather than a type for one field.

The obvious dirty solution is to deserialise the received json object twice, in the first round only for dataType and in the second round for all fields with the correct DTO, but of course I'd prefer something that would at least parse the json object only once (and map twice).

junyama

@junyama

This message was deleted

1 reply

axelspoerl

@axelspoerl

Hi, I am creating async listeners with the following code:

OATPP_COMPONENT(std::shared_ptr<std::list<std::shared_ptr<oatpp::network::ServerConnectionProvider>>>, connectionProviders);

std::list<std::thread> threads;

for(auto& provider : *connectionProviders) {
threads.push_back(std::thread([provider]{
OATPP_COMPONENT(std::shared_ptr<oatpp::network::ConnectionHandler>, handler);
oatpp::network::Server server(provider, handler);
server.run();
}));
}

std::thread statThread([]{
printStats();
});

My question is: How do I properly shut the listeners down?

Zach

@acidtonic

If you look at the docs they cover the various start/stop configurations depending on if you need to stop later or not.

oat++

@oatpp_io_twitter

@mmn:matrix.org
So based on the value of dataType, I would need to select a whole DTO rather than a type for one field.
The obvious dirty solution is to deserialise the received json object twice, in the first round only for dataType and in the second round for all fields with the correct DTO, but of course I'd prefer something that would at least parse the json object only once (and map twice).

Yes, in this case, you just have to parse and map it twice.

@axelspoerl
See this example project - https://github.com/oatpp/example-server-stop - it's not yet updated to 1.3.0 but API is almost the same

junyama

@junyama

I have set RetryPolicy as follows:

std::unordered_set<v_int32> m_httpCodes{500};
auto retryPolicy = std::make_shared<oatpp::web::client::SimpleRetryPolicy>(10, std::chrono::seconds(1), m_httpCodes);

Looking at the log of my client code, I saw some strange HTTP response status code, such as 3 and 4. My ENDPOINT is based on CRUD example:

ENDPOINT("POST", "images", createImage, BODY_DTO(Object<ImageDto>, imageDto)) {
      auto userDto = UserDto::createShared();
      serializeImageDto(imageDto, userDto);
      return createDtoResponse(Status::CODE_200, m_imageServices.createUser(userDto));
}

I do not change UserService.cpp. What was wrong? Thank you for your help in advance.

shijiantouzouyiqie

@shijiantouzouyiqie

@oatpp_io_twitter
Hello, question: when the server takes the initiative to send data, F5 refreshes the browser page. The server suddenly reports an error. What's the better way?

Thank you. Here is an example:

class WSInstanceListener : public oatpp::websocket::ConnectionHandler::SocketInstanceListener {
public:
    static bool m_bState;
    ...

};

bool WSInstanceListener::m_bState = true;
void WSInstanceListener::onAfterCreate(const oatpp::websocket::WebSocket& socket, const std::shared_ptr<const ParameterMap>& params) {
  SOCKETS ++;
  OATPP_LOGD(TAG, "New Incoming Connection. Connection count=%d", SOCKETS.load());

  /* In this particular case we create one WSListener per each connection */
  /* Which may be redundant in many cases */
  socket.setListener(std::make_shared<WSListener>());

  m_bState = true;

  std::thread t([&socket]{
    while (m_bState) {
        socket.sendOneFrameText("Hello");
    }
  });

  t.detach();
}

void WSInstanceListener::onBeforeDestroy(const oatpp::websocket::WebSocket& socket) {

  SOCKETS --;
  OATPP_LOGD(TAG, "Connection closed. Connection count=%d", SOCKETS.load());

  m_bState = false;
}

 I |2021-12-09 18:45:59 1639043159419263| MyApp:Server running on port 8000
 D |2021-12-09 18:46:06 1639043166551007| Server_WSListener:onClose code=1001
 D |2021-12-09 18:46:06 1639043166551045| Server_WSInstanceListener:Connection closed. Connection count=0
terminate called after throwing an instance of 'std::runtime_error'
  what():  [oatpp::web::protocol::websocket::WebSocket::sendOneFrameText()]: Unknown error while writing to socket.

oat++

@oatpp_io_twitter

Hello @junyama ,

Please give me more information:

exact errors that you see
client code - how you make the request

17 replies

Hello, question: when the server takes the initiative to send data, F5 refreshes the browser page. The server suddenly reports an error. What's the better way?

Hey @shijiantouzouyiqie , if you refresh the page in web-browser then the client is disconnected from the server, and the server reports that it can't use that already invalid connection.

1 reply

shijiantouzouyiqie

@shijiantouzouyiqie

@oatpp_io_twitter
Hello,after refreshing the browser, the program crashes. How to optimize the error point whenwsinstancelistener::onaftercreate uses std::threadto actively send data?
Or doesstd::threadnot cause the program to crash!

Mo Aboulmagd

@Maboulmagd

Hi I was wondering how to deserialize the following using oatpp::DTO:

"changes": [
    [
      "buy",
      "9122.04",
      "0.00121425"
    ],
    ...,
    [
      "sell",
      "9122.07",
      "0.98942292"
    ]
    ...
  ],

I had hoped I'd be able to do something like this:
Vector<Object<Change>> as opposed to Vector<Vector<String>>>. Ideas?

Also wondering how to do something similar for this:

 "trades": [
      {
          "type": "trade",
          "symbol": "BTCUSD",
          "eventid": 169841458,
          "timestamp": 1560976400428,
          "price": "9122.04",
          "quantity": "0.0073173",
          "side": "sell"
      },
      ...
  ],

Thanks.

Mo Aboulmagd

@Maboulmagd

On another note, what is the advantage of using oatpp DTO's over nlohmann::json?

Zach

@acidtonic

lol I used to work at Bloomberg, so many memories seeing stock data again :)

there exists a special database tailor made for this stuff and they open sourced it... Check out comdb2 which a clusterable sqlite lightweight db optimized for stock data

oat++

@oatpp_io_twitter

Hey @shijiantouzouyiqie ,

Hello,after refreshing the browser, the program crashes. How to optimize the error point whenwsinstancelistener::onaftercreate uses std::threadto actively send data?
Or doesstd::threadnot cause the program to crash!

You can see the error, right? Try to use try - catch

terminate called after throwing an instance of 'std::runtime_error'
  what():  [oatpp::web::protocol::websocket::WebSocket::sendOneFrameText()]: Unknown error while writing to socket.

Also, I wouldn't recommend using my example "as is". It's better to at least pool your threads.

Hey @Maboulmagd ,

Hi I was wondering how to deserialize the following using oatpp::DTO:

"changes": [
    [
      "buy",
      "9122.04",
      "0.00121425"
    ],
    ...,
    [
      "sell",
      "9122.07",
      "0.98942292"
    ]
    ...
  ],

Vector<Vector<String>>>

 "trades": [
      {
          "type": "trade",
          "symbol": "BTCUSD",
          "eventid": 169841458,
          "timestamp": 1560976400428,
          "price": "9122.04",
          "quantity": "0.0073173",
          "side": "sell"
      },
      ...
  ],

Vector<Object<MyObj>>

oat++

@oatpp_io_twitter

On another note, what is the advantage of using oatpp DTO's over nlohmann::json?

One should use whatever serves the need better.

Mo Aboulmagd

@Maboulmagd

@oatpp_io_twitter Makes sense, and thank you, it seems as though I was just not seeing that the data I was getting was mostly strings, as opposed to ints/floats, so my custom DTO's were incorrect.
That said, I guess its worth benchmarking nlohmann::json vs mapper_->readFromString<T>, to see which approach is more performant?
@acidtonic I'm currently working for a trading firm, but this is just a side project for me to see how far I can go, and to learn more...That said, Bloomberg is actually a place I wouldn't mind working at, since its C++, though I do hear that its very team-dependent as to whether you use modern C++, or pre C++11. How was it working there, if you don't mind me asking? Also thanks for the database tip.

2 replies

Zach

@acidtonic

@Maboulmagd if you want to pm me I'll share some opinions.

1 reply

mmn

@mmn:matrix.org

[m]

For prices and quantities... the problem is that they are decimal values that don't necessarily directly map into floats/doubles.

There is an arbitrary-precision type NUMERIC in PostgreSQL, which is a good fit for this type of data. It is apparently not directly supported in oatpp-postgresql, but you can always pass strings to PostgreSQL and it will convert them to the correct types.

mmn

@mmn:matrix.org

[m]

(Well, NUMERIC is AFAIK in standard SQL, the problem is that standard SQL assumes that you specify a fixed decimal scale (PostgreSQL does not)... which might be a problem if you want to have price and quantity data for pairs like Shiba Inu/BTC on one hand, and BTC/Shiba Inu on the other.)

Nikola Radovanovic

@nikoladsp

hi all, short architectural question: say I want to create model as a shared library and use it in multiple micro-services (one service is one process), is it then possible to use single swagger page for all micro-services?

oat++

@oatpp_io_twitter

@nikoladsp ,

You can do that.

You can have a proxy in front of all your services which proxies requests + has the swagger-ui for all upstream services.
Or, you can serve swagger-ui in a separate service. Just make sure that you have the correct server address in swagger-ui and that all your services are accessible at that host-name

Nikola Radovanovic

@nikoladsp

@oatpp_io_twitter , many thanks. I will probably go with second approach

shijiantouzouyiqie

@shijiantouzouyiqie

The browser establishes a link with the server websocket and actively sends data. The browser refreshes the page and closes the page for the first time. The server runs normally; The browser establishes a link with the server websocket and actively sends data. When the browser closes the page for the second time, it will send 1001 status code to the server, and the program crashes;
@oatpp_io_twitter Hello, the following is the current code. Is there a better optimization method?

void* StartThread(void* arg);
class WSInstanceListener : public oatpp::websocket::ConnectionHandler::SocketInstanceListener {
public:

.....
public:

  pthread_t m_pid;
  bool m_bQuitThread;
  oatpp::websocket::WebSocket* m_Socket;

};

WSInstanceListener::WSInstanceListener() {
    m_pid = -1;
    m_bQuitThread = true;
    m_Socket = NULL;
}

void WSInstanceListener::onAfterCreate(
        const oatpp::websocket::WebSocket &socket,
        const std::shared_ptr<const ParameterMap> &params) {
    SOCKETS++;
    OATPP_LOGD(TAG, "New Incoming Connection. Connection count=%d",
            SOCKETS.load());

    socket.setListener(std::make_shared<WSListener>());

    m_Socket = (oatpp::websocket::WebSocket*) &socket;

    pthread_create(&m_pid, NULL, StartThread, (void*) this);
    pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL);
    pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
    pthread_detach(m_pid);

}

void WSInstanceListener::onBeforeDestroy(
        const oatpp::websocket::WebSocket &socket) {
    SOCKETS--;
    OATPP_LOGD(TAG, "Connection closed. Connection count=%d", SOCKETS.load());

    m_bQuitThread = false;
    pthread_cancel(m_pid);
    pthread_join(m_pid,NULL);
    this->m_Socket->stopListening();

}

void* StartThread(void *arg) {
    WSInstanceListener *lis = (WSInstanceListener*) arg;

    lis->m_bQuitThread = true;

    while (lis->m_bQuitThread) {
        pthread_testcancel();
        lis->m_Socket->sendOneFrameText("hello");
    }

    return (void*) 0;
}

 I |2021-12-15 19:46:53 1639568813189617| MyApp:Server running on port 8000
 D |2021-12-15 19:47:00 1639568820630942| Server_WSInstanceListener:New Incoming Connection. Connection count=1
 D |2021-12-15 19:47:06 1639568826556318| Server_WSListener:onClose code=1001
 D |2021-12-15 19:47:06 1639568826556353| Server_WSInstanceListener:Connection closed. Connection count=0
 D |2021-12-15 19:47:16 1639568836406451| Server_WSInstanceListener:New Incoming Connection. Connection count=1
 D |2021-12-15 19:47:21 1639568841210745| Server_WSListener:onClose code=1001
 terminate called after throwing an instance of 'std::runtime_error'
 D |2021-12-15 19:47:21 1639568841210782| Server_WSInstanceListener:Connection closed. Connection count=0
 terminate called recursively

AlexandreHURDYK

@AlexandreHURDYK

Hello.

I've got an annoying issue I'm trying to fix on my oat++ client application : if my server shuts down before it gets the time to send a response to my client after a client request, it causes my client to crash. This is probably a common problem. Any way to fix this client side ?

I can provide an example API call from my client if necessary.

There's probably something I forgot to do.

AlexandreHURDYK

@AlexandreHURDYK

Most basic example : if I call an API function while the server is not up, the client crashes.

As simple as that.

I can try to make a minimal working example.

Is there a way to make my request time out or something ?

Where communities thrive

People

Repo info

Activity