Bake walnuts at 350° for 6 minutes, stirring after 3 minutes. Set aside.
Combine honey, balsamic vinegar, and orange juice in a small saucepan. Bring mixture to a boil over high heat and cook for 3 minutes.
Once the walnuts have cooled, add to goat cheese and stir until evenly mixed.
Top whole grain crackers with a spoonful of the walnut and goat cheese mixture. Drizzle the honey vinegar mixture on top, and sprinkle with chives and pepper.