but you only need a static web server if you do it all at once. what is is that with gzip.. maybe like 8KB? honestly it's a clever way to minimize costs

Lol sounds like we're in violent agreement then

I'm specifically talking about hosting the data as a static file. Data collection is a different matter, but this thread has been talking about not needing to go through a database every time you want to see the pay breakdown.

I never said one word about using a spreadsheet as a database, so stop trying to put words in my mouth simply because can't defend your position on the need for the client to access a database.

Actually, their way scales beautifully, and your CRUD app is going to be a nightmare to scale. At the point where they truly have too much data, they can just publish the aggreagates and slightly modify their client code.

Look dude, I've been a software engineer for nearly 20 years now. I've worked on codebases larger than 1m lines, I've worked on projects spanning from low level embedded software to modern web apps, and everything in between. I've written code that processes 1TB of data per minute without using bloated frameworks like Spark. I know a clever solution, and that is indeed clever.

Pro tip: if you can't actually articulate why something should be done a certain way, then maybe you need to re-evaluate your assumptions. It sounds like you have a CRUD hammer and you think everything is a nail.

You want to hire a team of engineers to build a full blown service with all the coding, database maintenance, hosting, security, devops bullshit, etc. that entails to provide real time access to data that changes extremely slowly. Instead they got one dude to crank it out in a day and just update their public dataset with whatever new data has come in as a batch process. Their way is simpler, cheaper, and meets the needs of the user.

I'm not sure how they're actually collecting the data in this scenario. I imagine they don't just take submitted salaries as truth without some further verification steps, since relying on the honor system has been a losing internet strategy since 1993.. So given that, I'm going to say that real-time access to non-real-time data is kinda pointless.

If you can do it as a static page, there's no reason not to. Over-engineering is bad engineering.

Eh, depends on how they arranged the data and if they included anything beyond what was actually necessary.. I assume they already broke it down into a pivot on company name and title. 6-7 bytes each for the salary 1 for each comma, multiplied by 2 to include the stock bonus, and maybe a 50% premium for the company name , position name, and assorted brackets. that would bring us down to ~20 bytes per data point, maybe 2MB uncompressed... gzip can usually get 95% compression or better on text with lots of repeated characters, so maybe 100KB as a conservative estimate. With rounding to the nearest 1000 and a bit of luck on compression it could be as low as 20KB. That is higher than my naive guess of 8KB, but also probably less than the various jpegs and such that also load as part of the site.