Your issue is that you're using the default (old) binding to GDAL, based on Fiona [0].
You need to use pyogrio [1], its vectorized counterpart, instead. Make sure you use `engine="pyogrio"` when calling `to_file` [2]. Fiona does a loop in Python, while pyogrio is exclusively compiled. So pyogrio is usually about 10-15x faster than fiona. Soon, in pyogrio version 0.8, it will be another ~2-4x faster than pyogrio is now [3].
CSV is still faster than geo-formats with pyogrio. From what I saw, it writes most of the file quickly, then spends a lot of time, I think, building the index.
> %%timeit -n 1
> d.to_csv('/tmp/test.csv')
10.8 s ± 1.05 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
> %%timeit -n 1
> d2.to_file('/tmp/test.gpkg', engine='pyogrio')
1min 15s ± 5.96 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
> %%timeit -n 1
> d.to_csv('/tmp/test.csv.gz')
35.3 s ± 1.37 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
> %%timeit -n 1
> d2.to_file('/tmp/test.fgb', driver='FlatGeobuf', engine='pyogrio')
19.9 s ± 512 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
> ls -lah /tmp/test*
-rw-rw-r-- 1 culebron culebron 228M мар 27 11:02 /tmp/test.csv
-rw-rw-r-- 1 culebron culebron 63M мар 27 11:27 /tmp/test.csv.gz
-rw-rw-r-- 1 culebron culebron 545M мар 27 11:52 /tmp/test.fgb
-rw-r--r-- 1 culebron culebron 423M мар 27 11:14 /tmp/test.gpkg
That's why I'm working on the GeoParquet spec [0]! It gives you both compression-by-default and super fast reads and writes! So it's usually as small as gzipped CSV, if not smaller, while being faster to read and write than GeoPackage.
Try using `GeoDataFrame.to_parquet` and `GeoPandas.read_parquet`
You need to use pyogrio [1], its vectorized counterpart, instead. Make sure you use `engine="pyogrio"` when calling `to_file` [2]. Fiona does a loop in Python, while pyogrio is exclusively compiled. So pyogrio is usually about 10-15x faster than fiona. Soon, in pyogrio version 0.8, it will be another ~2-4x faster than pyogrio is now [3].
[0]: https://github.com/Toblerity/Fiona
[1]: https://github.com/geopandas/pyogrio
[2]: https://geopandas.org/en/stable/docs/reference/api/geopandas...
[3]: https://github.com/geopandas/pyogrio/pull/346