It will hurt you, a lot. You need to get at least 3 Rad Onc letters of recommendation (LOR's), hopefully from well known people, ideally you should have 4 all from well known people. They help more than you could ever imagine, seriously its amazing how much stoke residency programs put in them. Amazing. LOR's are definitely in the top 3 most important factors for matching well (which are namely boards, lors, and research)
They are pretty easy to get, just do an away rotation at any big Rad Onc program and try to work with the chair or another well known faculty member there. If you are at least on time and a generally interested/nice person they will likely write you a letter. Even if you only work with them a couple of times, they will still likely write you a letter. Good luck.