Preheat the oven to 400°F (204°C).
Prepare the marinade: in a bowl whisk 2 tablespoons oil, 2 tablespoons Dijon mustard, 3 chopped garlic cloves, 1 chopped shallot, chopped rosemary, 1 teaspoon smoked paprika, and a pinch each of kosher salt and black pepper.
Toss the chicken with the marinade until evenly coated and set aside to rest while you heat the skillet.
Heat 1 tablespoon olive oil in a large oven-safe skillet over medium-high heat. Sear the chicken until golden, about 3–5 minutes per side.
During the last 2 minutes of searing, add 3 tablespoons butter and the lemon slices to the pan so they brown slightly, then remove the chicken and lemon slices to a plate, reserving pan juices.
Add the orzo, the remaining chopped shallot, and 1 clove minced or grated garlic to the skillet and cook, stirring, until the orzo is lightly toasted, about 2 minutes.
Pour in 3 cups low-sodium chicken broth, add the chopped kale, and 2 tablespoons lemon juice; season with salt and pepper and bring to a boil over high heat.
Slide the seared chicken and lemon slices back into the skillet along with any collected juices. Transfer the skillet to the preheated oven and bake uncovered for about 15 minutes, or until the chicken is cooked through and the orzo is tender.
While the skillet bakes, make the feta sauce: combine 4–6 ounces feta, 1/4 cup plain Greek yogurt, 1 grated garlic clove, juice from 1 lemon, 1/4 teaspoon smoked paprika, and a pinch of crushed red pepper flakes in a blender and blend until smooth; thin with extra lemon juice if desired.
Remove the skillet from the oven. Serve the chicken and orzo topped with the creamy feta sauce and sprinkled with chopped fresh dill.
