My house came with a Nest installed and I think it looks cool so I left it in but never gave it the wifi password and turned off absolutely all the crazy "smart" features it had. Now it's a really neat looking dumb thermostat and I want nothing else.
My house also came with a nest, and I also never connected it to my network, but for me nest was a terrible dumb thermostat.
It constantly tried to infer schedules and change the temperature on its own. I would set the temperature, come back an hour later to find that it changed itself back to what it thought it should be.
Also, there was no way to just activate the fan. I live in a very temperate climate and I generally like to keep a few windows open but run the fan to circulate air through the house.
I sold the nest and now a $15 dumb thermostat from the local hardware store now lets me set a temperature and it won't randomly change it when it feels like it. And it has a switch to turn on the fan.
I managed to disable the scheduling system, as even that was too smart for me (I had similar troubles as you). All it does now is: set temperature; allow switching between heat, cold, both, or none; allow turning on fan-only for X hours. Maybe yours wasn't wired correctly for the fan? No idea.