"石竹科的" can be translated as "Caryophyllaceous" or more commonly and precisely in botanical contexts as "of the Caryophyllaceae family" (referring to the plant family Caryophyllaceae, which is the standard English name for 石竹科).
If you're specifically referring to something belonging to this family in a concise way, "Caryophyllaceous" might be used in technical or scientific writing, but "of the Caryophyllaceae family" is clearer and more widely understood.
For example:
"石竹科的植物" can be translated as "plants of the Caryophyllaceae family."