Problems
Small plants can become swamped by gravel. Plant them on a low mound so that that they are raised slightly above the level of the gravel.
Where landscape fabric is not used, weeds and self-seeding plants may become problematic, popping up where not wanted. However, allowing self-seeding can give the garden a more natural look even if some weeding is required.
If your soil type is heavy, clay-based, damp, or has a high water table, then gravel garden plants may not thrive. You may be better off going for a different style of garden and selecting planting to suit the conditions.