This post has been de-listed
It is no longer included in search results and normal feeds (front page, hot posts, subreddit posts, etc). It remains visible only via the author's post history.
My usecase involves parsing and processing very large streams of binary data and distilling a smaller aggregated summary out of this. At my workplace C is often used for this, but I wonder if there are FP languages that would be a good fit for this. Especially because pure FP should in theory make it easier to parallellize.
Constant factor optimizations are what you do only when you've exhausted every other avenue.. In a recent project I did using Azure Batch I was getting a rate of $0.02/hour for single core nodes (half that if you use low priority nodes). For 1000 nodes, that's $20/hour. Meanwhile I make about $150/hour.. It would take a lot of compute time to make further optimization worthwhile.
I use Haskell to process about 2 TB of data every day, which takes around 10 minutes... I could maybe double or triple the performance using C, but as it turns out engineering time is expensive and compute time is cheap.
Subreddit
Post Details
- Posted
- 2 years ago
- Reddit URL
- View post on reddit.com
- External URL
- reddit.com/r/functionalp...
Sure, as always it's best to check the xkcd chart. In this case you were initially responding to someone using Haskell instead of C for big data workloads though, and my point is that switching to low level programming to squeeze out 2-3x gains is almost never worth it.