A place to discuss and ask questions about using Scala for Spark programming.
Thread
s, making connections or reading files, all bets are off. But I suspect it is not.
Hi I am trying to copy a query into spark sql. Does the syntax below make sense. I am just constantly getting an error and can't fix it. I took in two data tables from a database and joined them together in one and created a temp view referred to as temp... so the from temp is the temp view that I created
SELECT
case when RESULT_CD is not null then substr(RESULT_CD,1,1)
else
case when RESULT_RSN_CD is not null then RESULT_RSN_CD else null end
end
as LOSSCD, LGL_SYS_ID as LGL_SYS_ID
FROM (
SELECT
LGL_SYS_ID,
MATTER_SYS_ID,
row_number() OVER (PARTITION BY LGL_SYS_ID ORDER BY
(cast(substr(MATTER_SYS_ID,instr(MATTER_SYS_ID,':')1)+) DESC) as int) rank ,
FTRSYS_ID, RESULT_RESLTN_CD, RESULT_RSN_CD
FROM TEMP
WHERE HCO_MATTER.LGL_MATTER_SYS_ID = HCO_FTR.FTR_SYS_ID
ORDER BY LGL_MATTER_HCS_SYS_ID desc)
WHERE rank=1 --
Hey everyone, I'm trying something which is not working and was hoping I could get some advice. I have a column which I am trying to use the actual value of to select from another column based off of that value. I.e. my first column A is a 3 letter month description like "DEC", I want to use that to select from the column "DEC_VAL", or if it is "NOV" I want to select from "NOV_VAL".
So far I've got it to the point that I can correctly concat and all of that to get the right column name - but I can't figure out how to select from that column given its name. I'm trying a lot of variants like:
.withColumn(
"othercol",
lit(col("mycol"))
)
but I can't seem to make it figure out how to use the literal value of that column to select the column name. Is this just chasing ghosts or is there some way to do this that I'm just not aware of?
InternalRow
s (specifically GenericInternalRow
),Analyzer
to resolve my LogicalPlan
references, because I can’t create a CatalogManager
which is private[sql]
,