The issue, though, is that this necessitates an essential equality of roles between individuals within the Body of Christ. Some are hands, others feet. Some are diakonoi, some presbyteroi, and some episkopoi (deacons, priests, and bishops), while others have no official ecclesial position, whom we call laity.
It is evidenced in both scripture and post-Scriptural early Church history that these roles are distinct and have differing responsibilities and authorities. Christ gave His chosen twelve the authority to forgive and retain sins (John 20), but not the broader group of followers. One may argue that "disciples" is ambiguous as to which ones (broader followers or the chosen twelve), but it is clarified in that the later passages of scripture do speak of confession in the context priestly duties, not general.
Further, the word in James 5 often translated "elders" is "πρεσβυτέρους," or "presbyteros", and so implies the confession in the following verses is to a priest (verses 13-18 has about enough context). In fact, the Douay-Rheims does translate it "priests."
I also should point out 2 Cor 2:10 and 2 Cor 5:18-20 which say that Christ reconciles "through us," meaning the "ambassadors for Christ," or those Christ sent, aka Apostles. Ministries need ministers, who "in the person of Christ" reconcile others to Him. We can see from the above passages that the power to forgive is not only with the Apostles* alone but also extends to the priests (presbyters).
Finally, one who carefully examines these still may find a very fine ambiguity about whether the power to forgive extends beyond priests, but positive confirmation of such is entirely lacking, including any inexplicit implied, neither to deacons nor laity, and post-Scripture Church history shows that they did not believe so.
Josh is right that it was initially a public thing, usually at the beginning of worship. I didn't know the origin of private confession, so that's interesting.
* The Apostles are the first bishops, see Act 1:20, "office" of an Apostle is referred to as ἐπισκοπὴν (episkopon), or bishopric.