It'd be really simple, really. Stick with what you know and merely hint at everything else.
I mean, that's all Rowling did to begin with. Stick with a British Boarding School model, invent what you need, mine inconsistencies/vague things in later installments and drop other aspects entirely.
With that imo lazy world building, she created what attracted folks to the series: a sense of wonder and discovery.
World building wise, I think the fandom puts in waaay more effort than Rowling ever did.