npgsql/npgsql

Npgsql is the .NET data provider for Postgresql. It allows any program developed for .NET framework to access a Postgresql database server. It is implemented in 100% C# code. Postgresql versions since 9.0 are officially supported, others may work.

Jeremy Gamble

@GHosaPhat

I'm trying to teach myself how to use ANTLR and their plsql grammar to (hopefully) get where I want to be but, as I said, it would be a lot nicer if I didn't have to go through all of that (or could, at least, eventually replace that).

I would assume that Npgsql does some sort of parsing at some level behind the scenes. I'm certainly not saying I think this request is "simple", but I would guess that there's at least a partial framework in place for some of the task.

Jeremy Gamble

@GHosaPhat

Well, after a week of trying to get ANTLR to a place I could at least do some testing, I've given up for now. If a feature like I've described eventually becomes available in Npgsql (or if it's already there and I'm just not able to find it) or, if it's available in some other library somewhere that I haven't yet found, I'd be thrilled to hear about it. In the meantime, I'm just going to have to rebuild the SQL statements I need as I go.

I've posted a question on StackOverflow to this effect (.NET Library to Parse/Modify/Reconstruct SQL Statement) but, as I stated before, I'd certainly prefer to have it as a part of the Npgsql library I'm already using for most of my database communication anyway. If anyone else here has any suggestions or thoughts, I'd be glad to hear them. Thanks.

James

@pha3z

https://www.npgsql.org/doc/prepare.html shows an example of two UPDATE statements. It explains that the individual statements are prepared. However, it doesn't clearly state if using multiple statements with a single prepare in one command is actually the best practice. What if I want to do 200 Updates or 200 Inserts? I put all 200 Updates in a single parameterized query and call Prepare() once followed by ExecuteQuery(). Is this the optimal way to do it? Would it perform differently if I created 1 single Update statement in one query, prepared it, executed.... and then put the remaining 199 in a second query without preparing it? I feel like it would help if I understood better what's happening under the hood.

Another example on the same page demonstrates Preparing a query, Executing it, then changing the parameters and Executing it again (in a loop possibly). But would be a poor practice for something like a series of UPDATES or INSERTS, right? Changing parameters and executing costs round-trip delay. Maybe some guidance on how to use prepare based on what KIND of operation you're doing would be helpful.

James

@pha3z

@GHosaPhat SQL parsing is special problem all of its own. SQL syntax varies depends on the db provider and the db provider does its own parsing. Npgsql doesn't need to concern itself with the SQL structure. Adding a parser to npgsql would add unnecessary code to maintain. Npgsql is an ado.net compliant postgres driver and nothing more. It should remain that way

Jeremy Gamble

@GHosaPhat

@pha3z Thank you for the response. While I understand this position, as I've stated, having a parse tree available from within Npgsql would make my life so much easier for the project I'm currently working on. I guess it's back to the frustration of trying to learn how to use ANTLR in a VB.NET application to accomplish my goals. Thank you again.

James

@pha3z

@GHosaPhat What do you need to do the parsing for? And is this a commercial project?

@GHosaPhat Oh nevermind. I went back and read your original problem statement. You have a SQL statement already constructed and you need to modify some clauses in it

Jeremy Gamble

@GHosaPhat

Yep. That's it. ;)

Like I said, I can do it "manually", but I was looking for something a bit more "flexible" and reusable for a long-term solution.

James

@pha3z

Is this a commercial project and is there an opportunity to outsource the work? I am a consultant. I have my own text tokenizer and my own postgres ddl parser. It would not be a big deal for me to write you a sql parser for postgres

Jeremy Gamble

@GHosaPhat

It's not a commercial project in that it isn't something that our company will be selling to others. It's for an internal system that I've been building for our employees to use.

James

@pha3z

Alright. Well if there is an opportunity for consulting to assist with the sql parsing problem, let me know. I'd be glad to help.

Jeremy Gamble

@GHosaPhat

As for outsourcing, I wish I could say yes, but my budget is basically nil. I've paid for a couple of tools myself to do some testing on stuff (didn't ask for reimbursement b/c I wasn't sure if it would help).

James

@pha3z

i understand

Jeremy Gamble

@GHosaPhat

The "good news" is that I think I'm finally starting to make a little bit of progress with ANTLR, so maybe by 2025 I'll have something... :P

James

@pha3z

lol

mibiio

@mibiio

Hello, is there any way to modify the "DefaultValueType" of a mapping? Specifically I want to change the "DefaultValueType" of the ZonedDateTime mapping to "ZonedDateTime" to be able to retrive a ZonedDateTime with Dapper - currently it always returns an Instant as this is the default "DefaultValueType" that npgsql implements.

Ankkatalo

@Ankkatalo

Good evening. I'm looking for a .NET 6 version for prototyping, I guess I have to compile it myself, but what git branch or tag should I use as a basis?

Becky Conning

@beckyconning

Hi! What is the connection string, provider name and options that i need to use npgsql? thanks!

not sure even what a "provider name" is

Becky Conning

@beckyconning

ok so providername is Npgsql

now i'm doing {[CollectionName = "Tables"]}[Data]()and getting too few arguments error, will check docs.

Shay Rojansky

@roji

@GHosaPhat (and everyone else) sorry for answering so late, I was on vacation.

No, Npgsql doesn't come with a SQL parser - its job is only to take the SQL strings you provide it and send that to PostgreSQL. Parsing SQL is quite a complicated task as you've found out, especially if you start taking into accounts database-specific dialects...

@pha3z I think we covered preparation and batching in npgsql/doc#91

Jeremy Gamble

@GHosaPhat

OMG! You mean you aren't looking at code and issues 24/7 and actually took a little time for yourself? That's unacceptable! </sarcasm>

Shay Rojansky

@roji

I usually try to be very reactive, so answering this late is very annoying to me

@Ankkatalo you definitely don't need to compile .NET in order to use .NET 6 - previews are released on a monthly basis. Simply go to https://dotnet.microsoft.com/download/dotnet/6.0 and download the latest preview.

Becky Conning

@beckyconning

has anyone had experience using npgsql with power bi / m / power query? the connection seems to work but i get a table like the following and i'm not sure what to do next. what i'm expecting is a dialog that allows the user to select what tables they want.

CollectionName,NumberOfRestrictions,NumberOfIdentifierParts,Data,Function
MetaDataCollections,0,0,Table,Function
Restrictions,0,0,Table,Function
Databases,1,1,Function,Function
Tables,4,3,Function,Function
Columns,4,4,Function,Function
Views,3,3,Function,Function
Users,1,1,Function,Function
Indexes,4,4,Function,Function
IndexColumns,5,5,Function,Function

Shay Rojansky

@roji

@GHosaPhat FWIW EF Core internally uses a SQL expression tree to represent the queries as it builds them - and the PG provider extends that with PG-specific constructs and language elements. So at the end of the query pipeline you actually have a full expression tree representing the query, which is then rendered into text. However, this isn't parsing - it's constructing. If you find you need to parse a completed SQL in order to tweak it, it's obviously far better to go to the source which generated it and do your thing there. But I obviously know nothing about what you're doing and you probably can't for some reason.

@beckyconning that looks like the results for NpgsqlConnection.GetSchema... What exactly are you trying to do, which SQL query?

Becky Conning

@beckyconning

@roji i'd love to have a dialog that shows the available tables for the user to select

like the "Navigator" window that comes up for other sources

i am using the following AdoDotNet.DataSource("Npgsql", "Server=redacted;Port=5432;User Id=redacted;Password=redacted;Database=redacted")

Becky Conning

@beckyconning

are the functions' meanings and arguments detailed somewhere? for example if i call {[CollectionName = "Tables"]}[Data]() then i get a too few arguments (expecting 4) error.

Jeremy Gamble

@GHosaPhat

No, Npgsql doesn't come with a SQL parser - its job is only to take the SQL strings you provide it and send that to PostgreSQL. Parsing SQL is quite a complicated task as you've found out, especially if you start taking into accounts database-specific dialects...

I totally understand. I've kinda shelved that "side project" (parsing SQL statements) for now since I can't seem to get anything working correctly with ANTLR and I'm just having my application recreate the SQL statement completely instead of trying to manipulate its individual components.

That being said, for what I'm doing - extracting/parsing the SQL statement from a Crystal Reports document, updating one or more clauses, then push the modified SQL back into the document before the final report is generated/printed - it would still be nice to instead "simply" be able to access and modify that parse tree with the tools I'm already using (instead of banging my head against the ANTLR wall). As I said, I fully realize this is a fairly "one-off" type of usage situation in the grand scheme of things and there are other, "brute-force" methods of accomplishing the same goals.

Shay Rojansky

@roji

@beckyconning here's some code for getting the list of tables:

var tables = conn.GetSchema("Tables");
foreach (DataRow tableRow in tables.Rows)
{
Console.WriteLine(tableRow["TABLE_NAME"]);
}

Some docs can be found in https://docs.microsoft.com/en-us/dotnet/framework/data/adonet/sql-server-schema-collections#tables - it's the SQL Server docs but Npgsql tries to match the behavior

Beyond that I don't know anything about powerbi so I can't help much there...

Becky Conning

@beckyconning

thank you! : )

Alberto Passalacqua

@AlbertoPa

you were on vacation, don't feel annoyed. :-)

arjanvanvugt

@arjanvanvugt

Hello. Is there any way to get this provider available on a sql server instance to run queries against Postgres directly from sql server stored procedures?
Like you would do with a linked server based on the odbc connection, but then with this provider instead? (not allowed to use odbc). Help in any direction would be appreciated :)

Shay Rojansky

@roji

@arjanvanvugt IIRC there's an (old) way to have .NET user code embedded in SQL Server, to be executed from functions/procedures. You could use Npgsql from there somehow - nothing else comes to mind.

Ankkatalo

@Ankkatalo

When I said I'm looking for a .NET 6 version for prototyping, I meant a version of Npgsql.

arjanvanvugt

@arjanvanvugt

@roji Thanks. Yes like loading the dll with assemblies / clr ? I dont know exactly how that works either but after some investigation I dont think it will be allowed for us. Found there is a commercial ole db driver as well which can be used for linked servers. Think we will be using that. Thanks for your help!

Shay Rojansky

@roji

@Ankkatalo previews of Npgsql 6.0.0 are published roughly every month on nuget.org, here's the latest: https://www.nuget.org/packages/Npgsql/6.0.0-preview3

Ankkatalo

@Ankkatalo

Ah, excellent. Thanks.

Where communities thrive

People

Repo info

Activity