Use Variables in Legend Handles

I want to graph multiple different bar charts and the colors in each graph will vary depending on the data being plotted. I would like to have a legend with handles (to accompany each set of data and provide more detailed information) to make it clear what set of data each color represents.

ex: I may have two sets of data to plot. I can hardcode that easily (with some help from the patches library):

red_patch = mpatches.Patch(color ='red', label='red data')
blue_patch = mpatches.Patch(color ='blue', label='blue data')
ax.legend(handles=[red_patch, blue_patch])

But then in other cases I may want another color or even a few extra colors so the output could look like (in this case, don't need blue):

red_patch = mpatches.Patch(color ='red', label='red data')
blue_patch = mpatches.Patch(color ='blue', label='blue data')
yellow_patch = mpatches.Patch(color ='yellow', label='yellow data')
green_patch = mpatches.Patch(color ='green', label='green data')

ax.legend(handles=[red_patch, yellow_patch, green_patch])

I can define all the colors/handles but it's not really possible to hardcode every possible ax.legend call. That brings me to my question: Can I even use matplotlib handles dynamically?

ax.legend(handles=[])

I have a corresponding array that tells me which colors I want/need to show but I need a solution on how I can transfer that over for use in handles or a legend.