You can say that about all agriculture ever even more than you can about GMOs. We understand exact, precise genetic changes that have a much higher standards of testing than all of the genetic changes we've made to food using traditional techniques (mutagenesis, cross-breeding, selection) which modify dozens or hundreds of genes in highly unpredictable ways. We should be careful about all agricultural changes in a way proportional to what changes we made in the system, and not classify using modern science as always "too scary" to rely on.
If we're talking about Mendelevian use of alleles that exist in a species already, sure.
But have you seen those glow in the dark aquarium fish? Or the tomatoes that use flounder DNA to make them hardier in the freezer? How rigorously is it possible to test ramifications from those things?