spark parallelize data and its processing automagically, gives fault tolerance and checkpoining etc. You can implement it yourself, but it's a lot of work. Also, spark is JVM and it means that you have nearly bug free clients for all data storages, message brokers, kafka, etc, etc. and lots of libs. .NET is faaaar behind and I doubt it will ever catch up.