The problem seems simple on paper: there are specific, repeatable locations on the server where players can cross from one chunk to another and experience their game near-entirely freeze for a fraction of a second. As best as I can tell this happens when the server sends information regarding one (or both) of two chunks due to that player getting close enough to load said chunks. Because it seems centered on two chunks in particular, I'm inclined to believe it's the interaction between these two chunks, but it very well could be that both of those chunks have issues of their own and just happen to be side by side.

I've encountered this problem before several years ago, but at the time, it had to do with a large redstone-powered lighthouse that was causing havoc to players by creating frequent lighting updates right at the server's view distance. I've gone into spectator mode to observe the two chunks that are at the center of the problem, but I don't see anything that screams "this will be an absolute disaster" except for maybe the massive tree that is built there. The tree was build by placing a sapling, growing a tree with bone meal, topping the tree with dirt, and repeating up to nearly the build limit; it's a hell of a structure, but not something that I'd expect to be a problem... unless perhaps it has something to do with the leaves?

Either way, I'm kinda stumped. I'd love to eliminate the lag that happens any time a player crosses one of those borders, but at the moment, I have no idea what's even causing the issue. I plan on making a copy of the server to perform a few tests, but if anyone has a clue, I'd be happy to hear suggestions.

Thank you for your time.

EDIT: It's the damn tree.

I'm not entirely sure why, but clients seem to have one hell of a time processing sudden changes in maximum vertical elevation between chunks when said chunks are sent from the server to the client.

I originally thought perhaps the opaque nature of the leaves had something to do with it, so I built a platform over top of the tree to test that theory. I found it somehow only moved the boundary that was causing lag rather than eliminating it. I then decided to make a single line of dirt off of that platform into another chunk, and sure enough, it moved again. Even just placing a single dirt block at the vertical build limit in a chunk would cause that chunk to move where specifically the lag happened.

However, I wasn't sure if this meant it was pure height, or if it was height difference causing the issue, so I built a stairway of sorts. 16 blocks horizontal, 16 blocks down, repeat, all the way to the surface of the world. Following that, I knocked out nearly every block from this stairway, leaving only one per chunk--a single floating dirt block per 16 horizontal and vertical blocks, leading up to the sky limit.

With this structure, the boundary wasn't just moved... it was gone.

I've done a bit of testing, and here's what I'm confident about:

1. The problem occurs when the highest placed block in a chunk is significantly higher than the highest place block in adjacent chunks. I'm not sure what the actual difference needs to be before lag starts occurring, but my current assumption from my tests is that the larger the difference, the more intense the lag.

2. These problem chunks cause lag when to a client when the server sends information about that chunk to the client. This means that there is an X by X boundary around the problem chunk where lag can be experienced, with X being the server view distance (not the player's client view distance).

Interestingly, it seems the lag only occurs when the player moves back and forth across the boundary. More specifically, say the player is 20 chunks away from the problem chunk (player at chunk 0, problem at chunk 20), and is heading straight for it. Now say the lag can occur at the boundary of chunks 10 and 11. If the player moves from 10 to 11, and keeps moving onward, they won't experience it. If they move from 10, to 11, back to 10, back to 9, then forward again through 10 and 11, they still won't experience it. However, if they move from 10 to 11, then back to 10, then back 11, they will feel lag. This implies that there is a potential solution here that can be capitalized to fix the issue entirely.

I don't know if this has to do with how the server is sending information, or if the client is somehow predicting and avoiding a problem in one scenario but not the other, or what, but... I suspect if we're to solve this issue for good, this quirk of the lag hold some key piece of information that would be the solution to our problems.

As for how it can be fixed in the short term?

Well, I see two possible solutions, neither of which I love. One is to just tell players to avoid building extremely tall structures, or to at least make them slope gradually if they do. The other would be to place something very hard to see like a glass pane block (or maybe even a boundary block? at the sky limit, one per chunk, across the entire server, such that the highest block is always at the cap.

Either way, I hope this information helps. I'll continue to research solutions myself, though I'd love for anyone more knowledgeable on the inner workings of the client and server to take this information and perhaps implement a more permanent, widespread fix--or for anyone who might cross this thread to pass this information along to said people, since, well... I'm not sure who that would be, really.