The culling is noticeably slower than the camera movement. At first I didn't really think this was a "bug" per se, because I had over 8000 models in the scene; however, it is still noticeable (although to a lower extent) with model counts as low as 4. The models have between 0 and 1024 vertices (with most having 0).
I do this code for each model:
model->AddMesh(mesh);
model->UpdateBounds();
model->Staticize();
Here is a video of this happening with 8000 models (7600 of which have no vertices):
Another thing I tried was to filter out models with zero vertices from being created (I thought maybe that was the problem), but that didn't stop the issue (even for as little as 4 remaining models).